You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@poi.apache.org by Dominik Stadler <do...@gmx.at> on 2013/10/26 00:12:01 UTC

Tests for dev-tools like BiffViewer et.al.

Hi,

FYI, I just added a few tests which "bash" BiffViewer and other dev-tools
and verify that they do not fail with any of the test-files that we have
available in "test-data".

This ensures that existing files can still be read by these tools after
changes to the code and also that newly added test-files can be read by
these tools.

A few files currently fail the tests and are excluded to make the tests
pass, however I am not sure if all of these are ok to be excluded or if
some are actually code-errors that we should investigate. Can you let me
know which of the errors listed below indicate a coding-error and should be
investigated in more detail?

Also let me know comments/suggestions on adding such type of "bashing"
tests, I think it useful, but they are not strictly unit-tests any more...

Thanks... Dominik.

Failed: WORKBOOK_in_capitals.xls
Failed: NoGutsRecords.xls
Failed: BOOK_in_capitals.xls
Failed: OddStyleRecord.xls
java.io.FileNotFoundException: no such entry: "Workbook"
    at
org.apache.poi.poifs.filesystem.DirectoryNode.getEntry(DirectoryNode.java:375)
    at
org.apache.poi.poifs.filesystem.DirectoryNode.createDocumentInputStream(DirectoryNode.java:177)
    at
org.apache.poi.poifs.filesystem.POIFSFileSystem.createDocumentInputStream(POIFSFileSystem.java:445)
    at
org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:33)


Failed: password.xls
Failed: 50833.xls
Failed: 51832.xls
java.lang.IllegalArgumentException: Name is too long: ��fܓ��%
i\P���+��Z~u���]������miLÐ'$���HR�Yx)~ZzxId�hؤ�8��(
9�7O�VB}}q��u���Np��X�`���$X��[c
    at
org.apache.poi.hssf.record.WriteAccessRecord.setUsername(WriteAccessRecord.java:104)
    at
org.apache.poi.hssf.record.WriteAccessRecord.<init>(WriteAccessRecord.java:72)
    at org.apache.poi.hssf.dev.BiffViewer.createRecord(BiffViewer.java:253)
    at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:83)
    at org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
    at
org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)



Failed: XRefCalc.xls
java.lang.RuntimeException: Buffer overrun
    at
org.apache.poi.util.LittleEndianByteArrayInputStream.checkPosition(LittleEndianByteArrayInputStream.java:47)
    at
org.apache.poi.util.LittleEndianByteArrayInputStream.readUShort(LittleEndianByteArrayInputStream.java:100)
    at
org.apache.poi.util.LittleEndianByteArrayInputStream.readShort(LittleEndianByteArrayInputStream.java:93)
    at org.apache.poi.ss.formula.ptg.ExpPtg.<init>(ExpPtg.java:36)
    at org.apache.poi.ss.formula.ptg.Ptg.createBasePtg(Ptg.java:129)
    at org.apache.poi.ss.formula.ptg.Ptg.createPtg(Ptg.java:82)
    at org.apache.poi.ss.formula.ptg.Ptg.readTokens(Ptg.java:56)
    at org.apache.poi.ss.formula.Formula.getTokens(Formula.java:82)
    at
org.apache.poi.hssf.record.ExternalNameRecord.toString(ExternalNameRecord.java:212)
    at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:94)
    at org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
    at
org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)


Failed: 46904.xls
Failed: 43493.xls
org.apache.poi.hssf.record.RecordInputStream$LeftoverDataException:
Initialisation of record 0x10 left 8 bytes remaining still to be read.
    at
org.apache.poi.hssf.record.RecordInputStream.hasNextRecord(RecordInputStream.java:156)
    at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:67)
    at org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
    at
org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)
Discarding 8 bytes and continuing


Failed: 43493.xls
org.apache.poi.hssf.record.RecordFormatException: Expected size 18 but got
(0)
    at
org.apache.poi.hssf.record.CommonObjectDataSubRecord.<init>(CommonObjectDataSubRecord.java:86)
    at
org.apache.poi.hssf.record.SubRecord.createSubRecord(SubRecord.java:49)
    at org.apache.poi.hssf.record.ObjRecord.<init>(ObjRecord.java:93)
    at org.apache.poi.hssf.dev.BiffViewer.createRecord(BiffViewer.java:202)
    at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:83)
    at org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
    at
org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)


Failed: 49219.xlsjava.lang.NullPointerException
    at
org.apache.poi.ss.formula.constant.ConstantValueParser.getEncodedSize(ConstantValueParser.java:93)
    at
org.apache.poi.hssf.record.ExternalNameRecord.getDataSize(ExternalNameRecord.java:135)
    at
org.apache.poi.hssf.record.StandardRecord.getRecordSize(StandardRecord.java:32)
    at
org.apache.poi.hssf.model.InternalWorkbook.getSize(InternalWorkbook.java:1039)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.getBytes(HSSFWorkbook.java:1299)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.write(HSSFWorkbook.java:1210)
    at org.apache.poi.hssf.dev.ReSave.main(ReSave.java:56)
    at org.apache.poi.hssf.dev.TestReSave.runOneFile(TestReSave.java:32)


Failed: password.xls
Failed: 51832.xls
org.apache.poi.EncryptedDocumentException: Default password is invalid for
docId/saltData/saltHash
    at
org.apache.poi.hssf.record.RecordFactoryInputStream$StreamEncryptionInfo.createDecryptingStream(RecordFactoryInputStream.java:116)
    at
org.apache.poi.hssf.record.RecordFactoryInputStream.<init>(RecordFactoryInputStream.java:184)
    at
org.apache.poi.hssf.record.RecordFactory.createRecords(RecordFactory.java:440)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:314)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:277)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:211)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:355)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:336)
    at org.apache.poi.hssf.dev.ReSave.main(ReSave.java:42)
    at org.apache.poi.hssf.dev.TestReSave.runOneFile(TestReSave.java:32)


Failed: 49931.xls
java.lang.IllegalStateException: Found more than one PageSettingsBlock in
custom view settings sub-stream
    at
org.apache.poi.hssf.record.aggregates.CustomViewSettingsRecordAggregate.<init>(CustomViewSettingsRecordAggregate.java:51)
    at
org.apache.poi.hssf.model.InternalSheet.<init>(InternalSheet.java:174)
    at
org.apache.poi.hssf.model.InternalSheet.createSheet(InternalSheet.java:121)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:324)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:277)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:211)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:355)
    at
org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:336)
    at org.apache.poi.hssf.dev.ReSave.main(ReSave.java:42)
    at org.apache.poi.hssf.dev.TestReSave.runOneFile(TestReSave.java:35)

Had 301 files

Re: Tests for dev-tools like BiffViewer et.al.

Posted by Nick Burch <ap...@gagravarr.org>.
On Fri, 25 Oct 2013, Nirk Niggler wrote:
> If the CFB container format mandates this, and this container is also 
> used with DOC and PPT files, shouldn't the logic be contained in POIFS?

Quite possibly, but it would need to be a new method that's explicitly 
case-insensitive. We can't make changes to the current methods at this 
stage, without potentially breaking things for users. (Some people, eg 
those doing test tools or forensics stuff, really care about the case 
sensitivity)

Are you able to work up a patch for POIFS + NPOIFS to add case-insensitive 
fetching methods onto the interfaces + classes?

Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org

Re: Tests for dev-tools like BiffViewer et.al.

Posted by Dominik Stadler <do...@gmx.at>.
Thanks for the comments, I have now updated the tests and code to use the
actual logic from HSSFWorkbook for reading the record in all the dev-tools.
It is still not doing case-insensitive comparison, but rather tries a few
combinations (Workbook, WORKBOOK, BOOK), but this way we do whatever
HSSFWorkbook does and this makes most of the previously failing files work
fine.

Thanks... Dominik.


On Sat, Oct 26, 2013 at 1:06 AM, Nirk Niggler <ni...@gmail.com>wrote:

> If the CFB container format mandates this, and this container is also used
> with DOC and PPT files, shouldn't the logic be contained in POIFS?
>
>
> On Fri, Oct 25, 2013 at 4:02 PM, Nick Burch <ap...@gagravarr.org> wrote:
>
> > On Fri, 25 Oct 2013, Nirk Niggler wrote:
> >
> >> I haven't dug deep into the code, but a cursory glance of the POIFS code
> >> reveals no effort to convert the names to uppercase.
> >>
> >
> > Nope, POIFS doesn't do any name conversion. Things like HSSFWorkbook do
> > however
> >
> >
> > Nick
> >
> > ------------------------------**------------------------------**---------
> > To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
> > For additional commands, e-mail: dev-help@poi.apache.org
> >
> >
>

Re: Tests for dev-tools like BiffViewer et.al.

Posted by Nirk Niggler <ni...@gmail.com>.
If the CFB container format mandates this, and this container is also used
with DOC and PPT files, shouldn't the logic be contained in POIFS?


On Fri, Oct 25, 2013 at 4:02 PM, Nick Burch <ap...@gagravarr.org> wrote:

> On Fri, 25 Oct 2013, Nirk Niggler wrote:
>
>> I haven't dug deep into the code, but a cursory glance of the POIFS code
>> reveals no effort to convert the names to uppercase.
>>
>
> Nope, POIFS doesn't do any name conversion. Things like HSSFWorkbook do
> however
>
>
> Nick
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
> For additional commands, e-mail: dev-help@poi.apache.org
>
>

Re: Tests for dev-tools like BiffViewer et.al.

Posted by Nick Burch <ap...@gagravarr.org>.
On Fri, 25 Oct 2013, Nirk Niggler wrote:
> I haven't dug deep into the code, but a cursory glance of the POIFS code 
> reveals no effort to convert the names to uppercase.

Nope, POIFS doesn't do any name conversion. Things like HSSFWorkbook do 
however

Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org

Re: Tests for dev-tools like BiffViewer et.al.

Posted by Nirk Niggler <ni...@gmail.com>.
The first exception (java.io.FileNotFoundException: no such entry:
"Workbook") appears to stem from a POI bug:

Even though [MS-CFB] doesn't explicitly use the word "uppercase", the CFB
directory entries are definitely supposed to be treated as uppercase.
 v20130118 pages 27-28 describe the algorithm for comparing names:

> For each UTF-16 code point, convert to upper-case with the Unicode
Default Case Conversion Algorithm, simple case conversion variant ... Compare
each upper-cased UTF-16 code point binary value.

I haven't dug deep into the code, but a cursory glance of the POIFS code
reveals no effort to convert the names to uppercase.


On Fri, Oct 25, 2013 at 3:12 PM, Dominik Stadler <do...@gmx.at>wrote:

> Hi,
>
> FYI, I just added a few tests which "bash" BiffViewer and other dev-tools
> and verify that they do not fail with any of the test-files that we have
> available in "test-data".
>
> This ensures that existing files can still be read by these tools after
> changes to the code and also that newly added test-files can be read by
> these tools.
>
> A few files currently fail the tests and are excluded to make the tests
> pass, however I am not sure if all of these are ok to be excluded or if
> some are actually code-errors that we should investigate. Can you let me
> know which of the errors listed below indicate a coding-error and should be
> investigated in more detail?
>
> Also let me know comments/suggestions on adding such type of "bashing"
> tests, I think it useful, but they are not strictly unit-tests any more...
>
> Thanks... Dominik.
>
> Failed: WORKBOOK_in_capitals.xls
> Failed: NoGutsRecords.xls
> Failed: BOOK_in_capitals.xls
> Failed: OddStyleRecord.xls
> java.io.FileNotFoundException: no such entry: "Workbook"
>     at
>
> org.apache.poi.poifs.filesystem.DirectoryNode.getEntry(DirectoryNode.java:375)
>     at
>
> org.apache.poi.poifs.filesystem.DirectoryNode.createDocumentInputStream(DirectoryNode.java:177)
>     at
>
> org.apache.poi.poifs.filesystem.POIFSFileSystem.createDocumentInputStream(POIFSFileSystem.java:445)
>     at
> org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:33)
>
>
> Failed: password.xls
> Failed: 50833.xls
> Failed: 51832.xls
> java.lang.IllegalArgumentException: Name is too long: ��fܓ� �%
>  i\P��� +� �Z~u� ��]����� �miLÐ '$ ���HR�Yx) ~ZzxId�hؤ�  8��(
> 9�7O�VB}}q�  �u���Np��X�`���$X��[c
>     at
>
> org.apache.poi.hssf.record.WriteAccessRecord.setUsername(WriteAccessRecord.java:104)
>     at
>
> org.apache.poi.hssf.record.WriteAccessRecord.<init>(WriteAccessRecord.java:72)
>     at org.apache.poi.hssf.dev.BiffViewer.createRecord(BiffViewer.java:253)
>     at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:83)
>     at
> org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
>     at
> org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)
>
>
>
> Failed: XRefCalc.xls
> java.lang.RuntimeException: Buffer overrun
>     at
>
> org.apache.poi.util.LittleEndianByteArrayInputStream.checkPosition(LittleEndianByteArrayInputStream.java:47)
>     at
>
> org.apache.poi.util.LittleEndianByteArrayInputStream.readUShort(LittleEndianByteArrayInputStream.java:100)
>     at
>
> org.apache.poi.util.LittleEndianByteArrayInputStream.readShort(LittleEndianByteArrayInputStream.java:93)
>     at org.apache.poi.ss.formula.ptg.ExpPtg.<init>(ExpPtg.java:36)
>     at org.apache.poi.ss.formula.ptg.Ptg.createBasePtg(Ptg.java:129)
>     at org.apache.poi.ss.formula.ptg.Ptg.createPtg(Ptg.java:82)
>     at org.apache.poi.ss.formula.ptg.Ptg.readTokens(Ptg.java:56)
>     at org.apache.poi.ss.formula.Formula.getTokens(Formula.java:82)
>     at
>
> org.apache.poi.hssf.record.ExternalNameRecord.toString(ExternalNameRecord.java:212)
>     at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:94)
>     at
> org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
>     at
> org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)
>
>
> Failed: 46904.xls
> Failed: 43493.xls
> org.apache.poi.hssf.record.RecordInputStream$LeftoverDataException:
> Initialisation of record 0x10 left 8 bytes remaining still to be read.
>     at
>
> org.apache.poi.hssf.record.RecordInputStream.hasNextRecord(RecordInputStream.java:156)
>     at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:67)
>     at
> org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
>     at
> org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)
> Discarding 8 bytes and continuing
>
>
> Failed: 43493.xls
> org.apache.poi.hssf.record.RecordFormatException: Expected size 18 but got
> (0)
>     at
>
> org.apache.poi.hssf.record.CommonObjectDataSubRecord.<init>(CommonObjectDataSubRecord.java:86)
>     at
> org.apache.poi.hssf.record.SubRecord.createSubRecord(SubRecord.java:49)
>     at org.apache.poi.hssf.record.ObjRecord.<init>(ObjRecord.java:93)
>     at org.apache.poi.hssf.dev.BiffViewer.createRecord(BiffViewer.java:202)
>     at org.apache.poi.hssf.dev.BiffViewer.createRecords(BiffViewer.java:83)
>     at
> org.apache.poi.hssf.dev.BiffViewer.runBiffViewer(BiffViewer.java:427)
>     at
> org.apache.poi.hssf.dev.TestBiffViewer.runOneFile(TestBiffViewer.java:36)
>
>
> Failed: 49219.xlsjava.lang.NullPointerException
>     at
>
> org.apache.poi.ss.formula.constant.ConstantValueParser.getEncodedSize(ConstantValueParser.java:93)
>     at
>
> org.apache.poi.hssf.record.ExternalNameRecord.getDataSize(ExternalNameRecord.java:135)
>     at
>
> org.apache.poi.hssf.record.StandardRecord.getRecordSize(StandardRecord.java:32)
>     at
>
> org.apache.poi.hssf.model.InternalWorkbook.getSize(InternalWorkbook.java:1039)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.getBytes(HSSFWorkbook.java:1299)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.write(HSSFWorkbook.java:1210)
>     at org.apache.poi.hssf.dev.ReSave.main(ReSave.java:56)
>     at org.apache.poi.hssf.dev.TestReSave.runOneFile(TestReSave.java:32)
>
>
> Failed: password.xls
> Failed: 51832.xls
> org.apache.poi.EncryptedDocumentException: Default password is invalid for
> docId/saltData/saltHash
>     at
>
> org.apache.poi.hssf.record.RecordFactoryInputStream$StreamEncryptionInfo.createDecryptingStream(RecordFactoryInputStream.java:116)
>     at
>
> org.apache.poi.hssf.record.RecordFactoryInputStream.<init>(RecordFactoryInputStream.java:184)
>     at
>
> org.apache.poi.hssf.record.RecordFactory.createRecords(RecordFactory.java:440)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:314)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:277)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:211)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:355)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:336)
>     at org.apache.poi.hssf.dev.ReSave.main(ReSave.java:42)
>     at org.apache.poi.hssf.dev.TestReSave.runOneFile(TestReSave.java:32)
>
>
> Failed: 49931.xls
> java.lang.IllegalStateException: Found more than one PageSettingsBlock in
> custom view settings sub-stream
>     at
>
> org.apache.poi.hssf.record.aggregates.CustomViewSettingsRecordAggregate.<init>(CustomViewSettingsRecordAggregate.java:51)
>     at
> org.apache.poi.hssf.model.InternalSheet.<init>(InternalSheet.java:174)
>     at
> org.apache.poi.hssf.model.InternalSheet.createSheet(InternalSheet.java:121)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:324)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:277)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:211)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:355)
>     at
> org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:336)
>     at org.apache.poi.hssf.dev.ReSave.main(ReSave.java:42)
>     at org.apache.poi.hssf.dev.TestReSave.runOneFile(TestReSave.java:35)
>
> Had 301 files
>

Re: Tests for dev-tools like BiffViewer et.al.

Posted by Nick Burch <ap...@gagravarr.org>.
On Sat, 26 Oct 2013, Dominik Stadler wrote:
> Failed: WORKBOOK_in_capitals.xls

I'd expect to work, especially if HSSFWorkbook is happy

> Failed: NoGutsRecords.xls

Might be expected to fail

> Failed: BOOK_in_capitals.xls

I'd expect to work, especially if HSSFWorkbook is happy

> Failed: OddStyleRecord.xls

Might be expected to fail?

> Failed: password.xls
> Failed: 50833.xls
> Failed: 51832.xls
> Failed: XRefCalc.xls
> Failed: 46904.xls
> Failed: 43493.xls
> Failed: 43493.xls
> Failed: 49219.xls
> Failed: 49931.xls

Not sure about these. Might be ok to fail. Can HSSFWorkbook read them?

(I'd expect anything that HSSFWorkbook can read to work fine with the dev 
tools, but we do also have some negative test files which probably won't 
behave properly with anything)

Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@poi.apache.org
For additional commands, e-mail: dev-help@poi.apache.org