You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pdfbox.apache.org by Waldemar Dick <wa...@skribble.com> on 2021/02/17 10:01:47 UTC
Signing always writes COS object for AcroForm and Pages, although
there are no changes
Dear all,
I have a problem when signing a PDF document twice, which already has signature form fields pre-placed on the document.
The problem is visible, when the document is signed the second time. The signatures are correct, but Acrobat says, that there was a change in the document structure before the second signature was created.
Message is something like this: “Form Fields with Property Changes > Field Person_1 on page 5”.
I know, that this doesn’t change the validity of the signature itself, but still it is not nice and people are insecure, what it exactly means.
Looking into the COS structure of the PDF, I see that there are more objects written (in the incremental save), than actually changed.
Two objects stand out:
a) the AcroForm dictionary
b) the Page dictionary
Both objects are the exactly the same from the beginning, but still are written with every signature applied.
I had a working workaround in the past, where I changed the AcroForm /Annots entry to a object reference. This worked some time ago, but not anymore.
We currently use PDFBox 2.0.19, I updated to 2.0.22 with no success.
Two questions:
a) Why are the AcroForm and Pages objects marked as changed, when there are no changes?
b) Can I somehow try to remove the “needToBeUpdated” flag, before a incremental save. I tried, but fail to find the right time to do so.
c) When I mark the COSArray referenced by AcroForm /Annots to "direct=false”, it is ignored in the COSWriter.
COSWriter#visitFromArray never checks, if the array is direct or not. But I am pretty sure, that this worked some time ago.
Happy about any pointers, what I can try next to fix the problem.
Kind regards,
Waldemar
Re: Signing always writes COS object for AcroForm and Pages, although
there are no changes
Posted by Tilman Hausherr <TH...@t-online.de>.
Thank you! I've added a source code comment and also improved the example.
Have a great weekend!
Tilman
Am 19.02.2021 um 16:36 schrieb Waldemar Dick:
> Thanks for all the help!
>
> The actual problem was quite trivial: the widget annotation of the
> signature field was missing the “print” annotation flag. There was no
> flag set at all.
> During signing PDFBox will set this flag and this caused the
> unexpected amount of updated objects in the incremental save.
>
> What PDFBox does is totally correct.
>
> If I set the flag on all widgets before signing, everything is fine.
>
>
> Regards,
> Waldemar
>
>
>> On 18. 02 2021, at 17:48, Tilman Hausherr <THausherr@t-online.de
>> <ma...@t-online.de>> wrote:
>>
>> What you could do if is to add this in COSWriter.prepareIncrement():
>>
>> PDAcroForm acroForm = doc.getDocumentCatalog().getAcroForm(null);
>> if (acroForm != null &&
>> !acroForm.getCOSObject().isNeedToBeUpdated())
>> {
>> // find fields that need to be updated, these may be
>> missed if there is no "path" to them
>> for (PDField field : acroForm.getFieldTree())
>> {
>> if (field.getCOSObject().isNeedToBeUpdated() &&
>> !objectsToWrite.contains(field.getCOSObject()))
>> {
>> objectsToWrite.add(field.getCOSObject());
>> }
>> }
>> }
>>
>> In PDDocument, find the line
>>
>> page.getCOSObject().setNeedToBeUpdated(true);
>>
>> and move this line below the line
>>
>> annotations.add(widget);
>>
>>
>> Also make sure that the first signing doesn't create a "certified"
>> PDF. That is what you get when calling SigUtils.setMDPPermission().
>>
>> Tilman
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
>> <ma...@pdfbox.apache.org>
>> For additional commands, e-mail: users-help@pdfbox.apache.org
>> <ma...@pdfbox.apache.org>
>>
>
>
Re: Signing always writes COS object for AcroForm and Pages, although
there are no changes
Posted by Waldemar Dick <wa...@skribble.com>.
Thanks for all the help!
The actual problem was quite trivial: the widget annotation of the signature field was missing the “print” annotation flag. There was no flag set at all.
During signing PDFBox will set this flag and this caused the unexpected amount of updated objects in the incremental save.
What PDFBox does is totally correct.
If I set the flag on all widgets before signing, everything is fine.
Regards,
Waldemar
> On 18. 02 2021, at 17:48, Tilman Hausherr <TH...@t-online.de> wrote:
>
> What you could do if is to add this in COSWriter.prepareIncrement():
>
> PDAcroForm acroForm = doc.getDocumentCatalog().getAcroForm(null);
> if (acroForm != null && !acroForm.getCOSObject().isNeedToBeUpdated())
> {
> // find fields that need to be updated, these may be missed if there is no "path" to them
> for (PDField field : acroForm.getFieldTree())
> {
> if (field.getCOSObject().isNeedToBeUpdated() && !objectsToWrite.contains(field.getCOSObject()))
> {
> objectsToWrite.add(field.getCOSObject());
> }
> }
> }
>
> In PDDocument, find the line
>
> page.getCOSObject().setNeedToBeUpdated(true);
>
> and move this line below the line
>
> annotations.add(widget);
>
>
> Also make sure that the first signing doesn't create a "certified" PDF. That is what you get when calling SigUtils.setMDPPermission().
>
> Tilman
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
> For additional commands, e-mail: users-help@pdfbox.apache.org
>
Re: Signing always writes COS object for AcroForm and Pages, although
there are no changes
Posted by Tilman Hausherr <TH...@t-online.de>.
What you could do if is to add this in COSWriter.prepareIncrement():
PDAcroForm acroForm = doc.getDocumentCatalog().getAcroForm(null);
if (acroForm != null &&
!acroForm.getCOSObject().isNeedToBeUpdated())
{
// find fields that need to be updated, these may be missed
if there is no "path" to them
for (PDField field : acroForm.getFieldTree())
{
if (field.getCOSObject().isNeedToBeUpdated() &&
!objectsToWrite.contains(field.getCOSObject()))
{
objectsToWrite.add(field.getCOSObject());
}
}
}
In PDDocument, find the line
page.getCOSObject().setNeedToBeUpdated(true);
and move this line below the line
annotations.add(widget);
Also make sure that the first signing doesn't create a "certified" PDF.
That is what you get when calling SigUtils.setMDPPermission().
Tilman
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@pdfbox.apache.org
For additional commands, e-mail: users-help@pdfbox.apache.org
Re: Signing always writes COS object for AcroForm and Pages, although
there are no changes
Posted by Waldemar Dick <wa...@skribble.com>.
Hi,
You are right. If I create an empty document with 2 signature fields and sign them both, everything works as expected.
I will look now into the signature field resources, maybe there is a change.
Thanks for the hint!
Waldemar
> On 17. 02 2021, at 19:15, Tilman Hausherr <TH...@t-online.de> wrote:
>
> Am 17.02.2021 um 18:39 schrieb Waldemar Dick:
>> Hi,
>>
>> After a couple hours of debugging, I understand how and why which objects are written.
>>
>> I use PDFBox for both signatures and started playing around with
>> org.apache.pdfbox.pdfwriter.COSWriter#COSWriter(java.io.OutputStream, org.apache.pdfbox.io.RandomAccessRead, java.util.Set<org.apache.pdfbox.cos.COSDictionary>)
>> without success.
>>
>> If I don’t write the AcroForm or Page, then Acrobat seams to not find the signature at all.
>
> I suspect it's not in the incremental part.
>
> I assume the problem is in the page... a possible solution would be to have COSWriter search for fields that have update flags, when in saveIncremental mode without the third parameter. I suspect that this field is missed per my "path" theory from the previous post.
>
> There is code in the example subproject that creates a PDF with an empty signature field, CreateEmptySignatureForm.java, maybe you can reproduce it with that one?
>
> Tilman
>
>
>
>
>
>> If I write them, then it shows a modification.
>>
>> I start having the feeling, that I am looking at the wrong place.
>>
>> Are there any flags, which could disallow further changes to the PDF document, even if these are just another signature?
>> Tried changing AcroForm /SigFlags from SIGNATURE_EXISTS and APPEND_ONLY to just SIGNATURE_EXISTS, which doesn’t change Acrobats behaviour.
>>
>> As always a debug log from Acrobat would be nice.
>>
>> For the PDF: I have to remove the content, before I can share it. Will try to do it as soon as possible. Right now, it is difficult to debug.
>>
>>
>> Waldemar
>>
>>
>>
>>
>>> On 17. 02 2021, at 18:02, Tilman Hausherr <THausherr@t-online.de <ma...@t-online.de>> wrote:
>>>
>>> Hi,
>>>
>>> page is set updated because the annotations array is updated and the array is always saved inside (not as an object).
>>>
>>> acroform is updated because fields is a direct object and fields is modified.
>>>
>>> Yes, it might make sense to fine-tune this, i.e. to set the update flag only if fields / annotations are added.
>>> The problem is that this code segment is very difficult, we have seen many weird problems (similar to yours!) that can't be discovered in automated tests.
>>>
>>> One risk I remember/suspect is that when we don't mark certain parts as updated, then dependent modified objects aren't updated either. There must be a "path" to the modified objects.
>>>
>>> Could you share the PDF files? (upload to sharehoster) Are you signing both with PDFBox or only one with it and the other with another product?
>>>
>>>
>>> Tilman
>>>
>>> Am 17.02.2021 um 11:01 schrieb Waldemar Dick:
>>>> Dear all,
>>>>
>>>> I have a problem when signing a PDF document twice, which already has signature form fields pre-placed on the document.
>>>>
>>>> The problem is visible, when the document is signed the second time. The signatures are correct, but Acrobat says, that there was a change in the document structure before the second signature was created.
>>>>
>>>> Message is something like this: “Form Fields with Property Changes > Field Person_1 on page 5”.
>>>>
>>>> I know, that this doesn’t change the validity of the signature itself, but still it is not nice and people are insecure, what it exactly means.
>>>>
>>>> Looking into the COS structure of the PDF, I see that there are more objects written (in the incremental save), than actually changed.
>>>>
>>>> Two objects stand out:
>>>> a) the AcroForm dictionary
>>>> b) the Page dictionary
>>>>
>>>> Both objects are the exactly the same from the beginning, but still are written with every signature applied.
>>>>
>>>> I had a working workaround in the past, where I changed the AcroForm /Annots entry to a object reference. This worked some time ago, but not anymore.
>>>> We currently use PDFBox 2.0.19, I updated to 2.0.22 with no success.
>>>>
>>>> Two questions:
>>>> a) Why are the AcroForm and Pages objects marked as changed, when there are no changes?
>>>>
>>>> b) Can I somehow try to remove the “needToBeUpdated” flag, before a incremental save. I tried, but fail to find the right time to do so.
>>>>
>>>> c) When I mark the COSArray referenced by AcroForm /Annots to "direct=false”, it is ignored in the COSWriter.
>>>>
>>>> COSWriter#visitFromArray never checks, if the array is direct or not. But I am pretty sure, that this worked some time ago.
>>>>
>>>>
>>>> Happy about any pointers, what I can try next to fix the problem.
>>>>
>>>>
>>>> Kind regards,
>>>> Waldemar
>>>
>>>
>>
>>
>>
>>
>>
>>
>>
>> Waldemar Dick
>> signing & security
>>
>> Mobile +49 (0)179 1106735
>> Support +41 (0)44 505 16 64
>> E-Mail waldemar@skribble.com <ma...@skribble.com>
>>
>> Pforzheimer Straße 128a, 76275 Ettlingen, Deutschland
>>
>> Qualified electronic signing made easy.
>> Skribble.com <https://www.skribble.com/>
>
Waldemar Dick
signing & security
Mobile +49 (0)179 1106735
Support +41 (0)44 505 16 64
E-Mail waldemar@skribble.com <ma...@skribble.com>
Pforzheimer Straße 128a, 76275 Ettlingen, Deutschland
Qualified electronic signing made easy.
Skribble.com <https://www.skribble.com/>
Re: Signing always writes COS object for AcroForm and Pages, although
there are no changes
Posted by Tilman Hausherr <TH...@t-online.de>.
Am 17.02.2021 um 18:39 schrieb Waldemar Dick:
> Hi,
>
> After a couple hours of debugging, I understand how and why which
> objects are written.
>
> I use PDFBox for both signatures and started playing around with
> org.apache.pdfbox.pdfwriter.COSWriter#COSWriter(java.io.OutputStream,
> org.apache.pdfbox.io.RandomAccessRead,
> java.util.Set<org.apache.pdfbox.cos.COSDictionary>)
> without success.
>
> If I don’t write the AcroForm or Page, then Acrobat seams to not find
> the signature at all.
I suspect it's not in the incremental part.
I assume the problem is in the page... a possible solution would be to
have COSWriter search for fields that have update flags, when in
saveIncremental mode without the third parameter. I suspect that this
field is missed per my "path" theory from the previous post.
There is code in the example subproject that creates a PDF with an empty
signature field, CreateEmptySignatureForm.java, maybe you can reproduce
it with that one?
Tilman
> If I write them, then it shows a modification.
>
> I start having the feeling, that I am looking at the wrong place.
>
> Are there any flags, which could disallow further changes to the PDF
> document, even if these are just another signature?
> Tried changing AcroForm /SigFlags from SIGNATURE_EXISTS and
> APPEND_ONLY to just SIGNATURE_EXISTS, which doesn’t change Acrobats
> behaviour.
>
> As always a debug log from Acrobat would be nice.
>
> For the PDF: I have to remove the content, before I can share it. Will
> try to do it as soon as possible. Right now, it is difficult to debug.
>
>
> Waldemar
>
>
>
>
>> On 17. 02 2021, at 18:02, Tilman Hausherr <THausherr@t-online.de
>> <ma...@t-online.de>> wrote:
>>
>> Hi,
>>
>> page is set updated because the annotations array is updated and the
>> array is always saved inside (not as an object).
>>
>> acroform is updated because fields is a direct object and fields is
>> modified.
>>
>> Yes, it might make sense to fine-tune this, i.e. to set the update
>> flag only if fields / annotations are added.
>> The problem is that this code segment is very difficult, we have seen
>> many weird problems (similar to yours!) that can't be discovered in
>> automated tests.
>>
>> One risk I remember/suspect is that when we don't mark certain parts
>> as updated, then dependent modified objects aren't updated either.
>> There must be a "path" to the modified objects.
>>
>> Could you share the PDF files? (upload to sharehoster) Are you
>> signing both with PDFBox or only one with it and the other with
>> another product?
>>
>>
>> Tilman
>>
>> Am 17.02.2021 um 11:01 schrieb Waldemar Dick:
>>> Dear all,
>>>
>>> I have a problem when signing a PDF document twice, which already
>>> has signature form fields pre-placed on the document.
>>>
>>> The problem is visible, when the document is signed the second time.
>>> The signatures are correct, but Acrobat says, that there was a
>>> change in the document structure before the second signature was
>>> created.
>>>
>>> Message is something like this: “Form Fields with Property Changes >
>>> Field Person_1 on page 5”.
>>>
>>> I know, that this doesn’t change the validity of the signature
>>> itself, but still it is not nice and people are insecure, what it
>>> exactly means.
>>>
>>> Looking into the COS structure of the PDF, I see that there are more
>>> objects written (in the incremental save), than actually changed.
>>>
>>> Two objects stand out:
>>> a) the AcroForm dictionary
>>> b) the Page dictionary
>>>
>>> Both objects are the exactly the same from the beginning, but still
>>> are written with every signature applied.
>>>
>>> I had a working workaround in the past, where I changed the AcroForm
>>> /Annots entry to a object reference. This worked some time ago, but
>>> not anymore.
>>> We currently use PDFBox 2.0.19, I updated to 2.0.22 with no success.
>>>
>>> Two questions:
>>> a) Why are the AcroForm and Pages objects marked as changed, when
>>> there are no changes?
>>>
>>> b) Can I somehow try to remove the “needToBeUpdated” flag, before a
>>> incremental save. I tried, but fail to find the right time to do so.
>>>
>>> c) When I mark the COSArray referenced by AcroForm /Annots to
>>> "direct=false”, it is ignored in the COSWriter.
>>>
>>> COSWriter#visitFromArray never checks, if the array is direct or
>>> not. But I am pretty sure, that this worked some time ago.
>>>
>>>
>>> Happy about any pointers, what I can try next to fix the problem.
>>>
>>>
>>> Kind regards,
>>> Waldemar
>>
>>
>
>
> *
>
>
> Waldemar Dick*
> signing & security
>
> *Mobile* +49 (0)179 1106735
> *Support* +41 (0)44 505 16 64
> *E-Mail***waldemar@skribble.com <ma...@skribble.com>
>
> Pforzheimer Straße 128a, 76275 Ettlingen, Deutschland
>
> *Qualified electronic signing made easy.*
> Skribble.com <https://www.skribble.com>
>
Re: Signing always writes COS object for AcroForm and Pages, although
there are no changes
Posted by Waldemar Dick <wa...@skribble.com>.
Hi,
After a couple hours of debugging, I understand how and why which objects are written.
I use PDFBox for both signatures and started playing around with
org.apache.pdfbox.pdfwriter.COSWriter#COSWriter(java.io.OutputStream, org.apache.pdfbox.io.RandomAccessRead, java.util.Set<org.apache.pdfbox.cos.COSDictionary>)
without success.
If I don’t write the AcroForm or Page, then Acrobat seams to not find the signature at all.
If I write them, then it shows a modification.
I start having the feeling, that I am looking at the wrong place.
Are there any flags, which could disallow further changes to the PDF document, even if these are just another signature?
Tried changing AcroForm /SigFlags from SIGNATURE_EXISTS and APPEND_ONLY to just SIGNATURE_EXISTS, which doesn’t change Acrobats behaviour.
As always a debug log from Acrobat would be nice.
For the PDF: I have to remove the content, before I can share it. Will try to do it as soon as possible. Right now, it is difficult to debug.
Waldemar
> On 17. 02 2021, at 18:02, Tilman Hausherr <TH...@t-online.de> wrote:
>
> Hi,
>
> page is set updated because the annotations array is updated and the array is always saved inside (not as an object).
>
> acroform is updated because fields is a direct object and fields is modified.
>
> Yes, it might make sense to fine-tune this, i.e. to set the update flag only if fields / annotations are added.
> The problem is that this code segment is very difficult, we have seen many weird problems (similar to yours!) that can't be discovered in automated tests.
>
> One risk I remember/suspect is that when we don't mark certain parts as updated, then dependent modified objects aren't updated either. There must be a "path" to the modified objects.
>
> Could you share the PDF files? (upload to sharehoster) Are you signing both with PDFBox or only one with it and the other with another product?
>
>
> Tilman
>
> Am 17.02.2021 um 11:01 schrieb Waldemar Dick:
>> Dear all,
>>
>> I have a problem when signing a PDF document twice, which already has signature form fields pre-placed on the document.
>>
>> The problem is visible, when the document is signed the second time. The signatures are correct, but Acrobat says, that there was a change in the document structure before the second signature was created.
>>
>> Message is something like this: “Form Fields with Property Changes > Field Person_1 on page 5”.
>>
>> I know, that this doesn’t change the validity of the signature itself, but still it is not nice and people are insecure, what it exactly means.
>>
>> Looking into the COS structure of the PDF, I see that there are more objects written (in the incremental save), than actually changed.
>>
>> Two objects stand out:
>> a) the AcroForm dictionary
>> b) the Page dictionary
>>
>> Both objects are the exactly the same from the beginning, but still are written with every signature applied.
>>
>> I had a working workaround in the past, where I changed the AcroForm /Annots entry to a object reference. This worked some time ago, but not anymore.
>> We currently use PDFBox 2.0.19, I updated to 2.0.22 with no success.
>>
>> Two questions:
>> a) Why are the AcroForm and Pages objects marked as changed, when there are no changes?
>>
>> b) Can I somehow try to remove the “needToBeUpdated” flag, before a incremental save. I tried, but fail to find the right time to do so.
>>
>> c) When I mark the COSArray referenced by AcroForm /Annots to "direct=false”, it is ignored in the COSWriter.
>>
>> COSWriter#visitFromArray never checks, if the array is direct or not. But I am pretty sure, that this worked some time ago.
>>
>>
>> Happy about any pointers, what I can try next to fix the problem.
>>
>>
>> Kind regards,
>> Waldemar
>
>
Waldemar Dick
signing & security
Mobile +49 (0)179 1106735
Support +41 (0)44 505 16 64
E-Mail waldemar@skribble.com <ma...@skribble.com>
Pforzheimer Straße 128a, 76275 Ettlingen, Deutschland
Qualified electronic signing made easy.
Skribble.com <https://www.skribble.com/>
Re: Signing always writes COS object for AcroForm and Pages, although
there are no changes
Posted by Tilman Hausherr <TH...@t-online.de>.
Hi,
page is set updated because the annotations array is updated and the
array is always saved inside (not as an object).
acroform is updated because fields is a direct object and fields is
modified.
Yes, it might make sense to fine-tune this, i.e. to set the update flag
only if fields / annotations are added.
The problem is that this code segment is very difficult, we have seen
many weird problems (similar to yours!) that can't be discovered in
automated tests.
One risk I remember/suspect is that when we don't mark certain parts as
updated, then dependent modified objects aren't updated either. There
must be a "path" to the modified objects.
Could you share the PDF files? (upload to sharehoster) Are you signing
both with PDFBox or only one with it and the other with another product?
Tilman
Am 17.02.2021 um 11:01 schrieb Waldemar Dick:
> Dear all,
>
> I have a problem when signing a PDF document twice, which already has
> signature form fields pre-placed on the document.
>
> The problem is visible, when the document is signed the second time.
> The signatures are correct, but Acrobat says, that there was a change
> in the document structure before the second signature was created.
>
> Message is something like this: “Form Fields with Property Changes >
> Field Person_1 on page 5”.
>
> I know, that this doesn’t change the validity of the signature itself,
> but still it is not nice and people are insecure, what it exactly means.
>
> Looking into the COS structure of the PDF, I see that there are more
> objects written (in the incremental save), than actually changed.
>
> Two objects stand out:
> a) the AcroForm dictionary
> b) the Page dictionary
>
> Both objects are the exactly the same from the beginning, but still
> are written with every signature applied.
>
> I had a working workaround in the past, where I changed the AcroForm
> /Annots entry to a object reference. This worked some time ago, but
> not anymore.
> We currently use PDFBox 2.0.19, I updated to 2.0.22 with no success.
>
> Two questions:
> a) Why are the AcroForm and Pages objects marked as changed, when
> there are no changes?
>
> b) Can I somehow try to remove the “needToBeUpdated” flag, before a
> incremental save. I tried, but fail to find the right time to do so.
>
> c) When I mark the COSArray referenced by AcroForm /Annots to
> "direct=false”, it is ignored in the COSWriter.
>
> COSWriter#visitFromArray never checks, if the array is direct or not.
> But I am pretty sure, that this worked some time ago.
>
>
> Happy about any pointers, what I can try next to fix the problem.
>
>
> Kind regards,
> Waldemar