You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "Steve Lawrence (Jira)" <ji...@apache.org> on 2023/02/02 16:36:00 UTC
[jira] [Commented] (DAFFODIL-2782) Abort: Invariant broken parsing large NITF file
[ https://issues.apache.org/jira/browse/DAFFODIL-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683487#comment-17683487 ]
Steve Lawrence commented on DAFFODIL-2782:
------------------------------------------
Looking at the code, this likely happens happens anytime a hex binary element has a length larger than 1073741823 bytes (~1 GB), which your length does. This is because we allocate a char array twice this size of the hex binary array since each byte becomes two chars, and doubling that size overflows into a negative number. I'm not sure there's really a way around this limit. Java has a maximum string length of 2GB, which means must have a hexBinary length of half that to represent it as a string. We should modify Daffodil to have an implementation maximum hexBinary length of 2GB / 2.
A workaround is to either switch to using a the Binary Large Object extension, or modify the schema so it parses the data as an array of smaller hexBinary elements instead of one giant one.
> Abort: Invariant broken parsing large NITF file
> -----------------------------------------------
>
> Key: DAFFODIL-2782
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2782
> Project: Daffodil
> Issue Type: Bug
> Affects Versions: 3.3.0
> Reporter: Lara Blatchford
> Priority: Major
>
> The test data would have to be approved for release by DISA. The test data request form and information about the files to be requested can be provided upon request.
> The exact value that causes the abort is unclear, but the following large integers occur in the file as seen using DISA's CIVA NITF viewer:
> NITF header file length: 002203434188
> 2 image segments each with length: 1101716399
>
> Exception in thread "main" org.apache.daffodil.exceptions.Abort: Invariant broken. Runtime.scala - Leaked exception: java.lang.NegativeArraySizeException
> java.lang.NegativeArraySizeException
> at org.apache.daffodil.util.Misc$.bytes2Hex(Misc.scala:313)
> at org.apache.daffodil.infoset.DISimple.dataValueAsString(InfosetImpl.scala:1348)
> at org.apache.daffodil.infoset.XMLTextInfosetOutputter.startSimple(XMLTextInfosetOutputter.scala:115)
> at org.apache.daffodil.japi.infoset.InfosetOutputterProxy.startSimple(Infoset.scala:480)
> at org.apache.daffodil.infoset.InfosetWalker.infosetWalkerStepMove(InfosetWalker.scala:455)
> at org.apache.daffodil.infoset.InfosetWalker.maybeDoStep(InfosetWalker.scala:297)
> at org.apache.daffodil.infoset.InfosetWalker.walk(InfosetWalker.scala:270)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parse(SequenceParserBases.scala:368)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.ElementParserBase.parse(ElementCombinator1.scala:135)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.Unseparated.parseOne(UnseparatedSequenceParsers.scala:29)
> at org.apache.daffodil.processors.parsers.Unseparated.parseOne$(UnseparatedSequenceParsers.scala:27)
> at org.apache.daffodil.processors.parsers.ScalarOrderedUnseparatedSequenceChildParser.parseOne(UnseparatedSequenceParsers.scala:37)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parseOneInstanceWithMaybePoU(SequenceParserBases.scala:438)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parseOneInstance(SequenceParserBases.scala:423)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parse(SequenceParserBases.scala:266)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.SpecifiedLengthParserBase.parse(SpecifiedLengthParsers.scala:76)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.ElementParserBase.parse(ElementCombinator1.scala:135)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.Unseparated.parseOne(UnseparatedSequenceParsers.scala:29)
> at org.apache.daffodil.processors.parsers.Unseparated.parseOne$(UnseparatedSequenceParsers.scala:27)
> at org.apache.daffodil.processors.parsers.ScalarOrderedUnseparatedSequenceChildParser.parseOne(UnseparatedSequenceParsers.scala:37)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parseOneInstanceWithMaybePoU(SequenceParserBases.scala:438)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parseOneInstance(SequenceParserBases.scala:423)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parse(SequenceParserBases.scala:266)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.ElementParserBase.parse(ElementCombinator1.scala:135)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.Unseparated.parseOne(UnseparatedSequenceParsers.scala:29)
> at org.apache.daffodil.processors.parsers.Unseparated.parseOne$(UnseparatedSequenceParsers.scala:27)
> at org.apache.daffodil.processors.parsers.RepOrderedExpressionOccursCountUnseparatedSequenceChildParser.parseOne(UnseparatedSequenceParsers.scala:55)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parseOneInstanceWithMaybePoU(SequenceParserBases.scala:438)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parseOneInstance(SequenceParserBases.scala:423)
> at org.apache.daffodil.processors.parsers.SequenceParserBase.parse(SequenceParserBases.scala:158)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.parsers.ElementParserBase.parse(ElementCombinator1.scala:135)
> at org.apache.daffodil.processors.parsers.Parser.parse1(Parser.scala:82)
> at org.apache.daffodil.processors.parsers.Parser.parse1$(Parser.scala:76)
> at org.apache.daffodil.processors.parsers.CombinatorParser.parse1(Parser.scala:161)
> at org.apache.daffodil.processors.DataProcessor.doParse(DataProcessor.scala:503)
> at org.apache.daffodil.processors.DataProcessor.parse(DataProcessor.scala:447)
> at org.apache.daffodil.japi.DataProcessor.parse(Daffodil.scala:812)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)