You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@daffodil.apache.org by "olabusayoT (via GitHub)" <gi...@apache.org> on 2024/10/15 22:07:32 UTC
[PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
olabusayoT opened a new pull request, #1338:
URL: https://github.com/apache/daffodil/pull/1338
- currently after trimming the value of the element, we set the valueLength, and then overwrite it after returning from the parse that does the trimming. This results in the wrong value for value length. This fixes it by checking if the valueLength has already been set, and only setting it in SpecifiedLengthParserBase.parse if it hasn't
- add tests
DAFFODIL-2658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1842574241
##########
daffodil-codegen-c/src/main/scala/org/apache/daffodil/codegen/c/DaffodilCCodeGenerator.scala:
##########
@@ -289,7 +289,8 @@ object DaffodilCCodeGenerator
case g: ElementParseAndUnspecifiedLength =>
elementParseAndUnspecifiedLengthGenerateCode(g, cgState)
case g: ElementUnused => noop(g)
- case g: HexBinaryLengthPrefixed => hexBinaryLengthPrefixedGenerateCode(g.e, cgState)
+ case g: HexBinaryEndOfBitLimit if g.e.isPrefixed =>
Review Comment:
Hmm...do you mean the suggestion to remove HexBinaryLengthPrefixed or the suggestion to rename all PrefixedLength parsers/unparser to BitLimitLength/MinimumLength?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1842555874
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1387,14 +1381,15 @@ trait ElementBaseGrammarMixin
}
case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
+ if isSimpleType &&
+ impliedRepresentation == Representation.Binary &&
+ primType != PrimType.HexBinary =>
new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
Review Comment:
So It look like nillable binary types need SpecifiedLengthImplicit., they are the only tests failing when I comment out that chunk of code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1941729436
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1322,60 +1328,75 @@ trait ElementBaseGrammarMixin
* as well, by not enclosing the body in a specified length enforcer.
*/
private def specifiedLength(bodyArg: => Gram) = {
- lazy val body = bodyArg
- lazy val bitsMultiplier = lengthUnits match {
- case LengthUnits.Bits => 1
- case LengthUnits.Bytes => 8
- case LengthUnits.Characters if knownEncodingIsFixedWidth => this.knownEncodingWidthInBits
- case _ => 0 // zero means can't multiply to get width in bits.
- }
- val lk = lengthKind
- lk match {
- case LengthKind.Delimited => body
- case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
- case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
- case LengthKind.Explicit => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthExplicitCharacters(this, body)
- }
- case LengthKind.Prefixed if (bitsMultiplier != 0) =>
- new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
- case LengthKind.Prefixed => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthPrefixedCharacters(this, body)
- }
- case LengthKind.Implicit
- if isSimpleType && primType == PrimType.String &&
- encodingInfo.knownEncodingIsFixedWidth => {
- //
- // Important case to optimize
- // If we can convert to a number of bits, then we should do so
- //
- val nBits = encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
- new SpecifiedLengthImplicit(this, body, nBits)
+ // we need this to evaluate before we wrap in specified length parser,
+ // so it can do any internal checks for example blobValue's check for
+ // non-explicit lengthKind
+ val body = bodyArg
+ if (
+ isSimpleType && impliedRepresentation == Representation.Text ||
+ isSimpleType && isNillable ||
+ isComplexType ||
+ lengthKind == LengthKind.Prefixed ||
+ isSimpleType && primType == PrimType.HexBinary && lengthKind == LengthKind.Pattern
+ ) {
+ lazy val bitsMultiplier = lengthUnits match {
+ case LengthUnits.Bits => 1
+ case LengthUnits.Bytes => 8
+ case LengthUnits.Characters if knownEncodingIsFixedWidth =>
+ this.knownEncodingWidthInBits
+ case _ => 0 // zero means can't multiply to get width in bits.
}
- case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
- new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
- case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
- new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
- case LengthKind.Implicit if isComplexType =>
- body // for complex types, implicit means "roll up from the bottom"
- case LengthKind.EndOfParent if isComplexType =>
- notYetImplemented("lengthKind='endOfParent' for complex type")
- case LengthKind.EndOfParent =>
- notYetImplemented("lengthKind='endOfParent' for simple type")
- case _ => {
- // TODO: implement other specified length like end of parent
- // for now, no restriction
- body
+ val lk = lengthKind
+ lk match {
+ case LengthKind.Delimited => body
+ case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
+ case LengthKind.Explicit if bitsMultiplier != 0 =>
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ case LengthKind.Explicit => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthExplicitCharacters(this, body)
+ }
+ case LengthKind.Prefixed if (bitsMultiplier != 0) =>
+ new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
+ case LengthKind.Prefixed => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthPrefixedCharacters(this, body)
+ }
+ case LengthKind.Implicit
+ if isSimpleType && primType == PrimType.String &&
+ encodingInfo.knownEncodingIsFixedWidth => {
+ //
+ // Important case to optimize
+ // If we can convert to a number of bits, then we should do so
+ //
+ val nBits =
+ encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
+ new SpecifiedLengthImplicit(this, body, nBits)
+ }
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
+ new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
+
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
+ new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
Review Comment:
hmm...im surprised it has no coverage, several tests pass that condition of isSimpleType and impliedRep == Text and isHexBinary and LK implicit!
ex: maxHexBinaryError, hexBinary_Implicit_03b etc
These tests fail when we remove that case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1941064848
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1322,60 +1328,75 @@ trait ElementBaseGrammarMixin
* as well, by not enclosing the body in a specified length enforcer.
*/
private def specifiedLength(bodyArg: => Gram) = {
- lazy val body = bodyArg
- lazy val bitsMultiplier = lengthUnits match {
- case LengthUnits.Bits => 1
- case LengthUnits.Bytes => 8
- case LengthUnits.Characters if knownEncodingIsFixedWidth => this.knownEncodingWidthInBits
- case _ => 0 // zero means can't multiply to get width in bits.
- }
- val lk = lengthKind
- lk match {
- case LengthKind.Delimited => body
- case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
- case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
- case LengthKind.Explicit => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthExplicitCharacters(this, body)
- }
- case LengthKind.Prefixed if (bitsMultiplier != 0) =>
- new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
- case LengthKind.Prefixed => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthPrefixedCharacters(this, body)
- }
- case LengthKind.Implicit
- if isSimpleType && primType == PrimType.String &&
- encodingInfo.knownEncodingIsFixedWidth => {
- //
- // Important case to optimize
- // If we can convert to a number of bits, then we should do so
- //
- val nBits = encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
- new SpecifiedLengthImplicit(this, body, nBits)
+ // we need this to evaluate before we wrap in specified length parser,
+ // so it can do any internal checks for example blobValue's check for
+ // non-explicit lengthKind
+ val body = bodyArg
+ if (
+ isSimpleType && impliedRepresentation == Representation.Text ||
+ isSimpleType && isNillable ||
+ isComplexType ||
+ lengthKind == LengthKind.Prefixed ||
+ isSimpleType && primType == PrimType.HexBinary && lengthKind == LengthKind.Pattern
+ ) {
Review Comment:
It might be worth setting this condition to a val to make it more self documenting. Even with that, it might also be worth adding comment clarifying why we do this. Maybe something like"
```scala
// there are essentially two categories of processors that read/write data input/output
// stream: those that calculate lengths themselves and those that expect another
// processor to calculate the length and set the bit limit which this processor will use as
// the length. The following determines if this element requires another processor to
// calculate and set the bit limit, and if so adds the appropriate grammar to do that
val bodyRequiresSpecifiedLengthBitLimit = isSimpleType && impliedRepresentation == Representation.Text || ...
if (!bodyRequiresSpecifiedLengthBitLimit) body
else lengthKind match {
// all the cases that add various specified length grammars that body depends on
}
```
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1322,60 +1328,75 @@ trait ElementBaseGrammarMixin
* as well, by not enclosing the body in a specified length enforcer.
*/
private def specifiedLength(bodyArg: => Gram) = {
- lazy val body = bodyArg
- lazy val bitsMultiplier = lengthUnits match {
- case LengthUnits.Bits => 1
- case LengthUnits.Bytes => 8
- case LengthUnits.Characters if knownEncodingIsFixedWidth => this.knownEncodingWidthInBits
- case _ => 0 // zero means can't multiply to get width in bits.
- }
- val lk = lengthKind
- lk match {
- case LengthKind.Delimited => body
- case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
- case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
- case LengthKind.Explicit => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthExplicitCharacters(this, body)
- }
- case LengthKind.Prefixed if (bitsMultiplier != 0) =>
- new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
- case LengthKind.Prefixed => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthPrefixedCharacters(this, body)
- }
- case LengthKind.Implicit
- if isSimpleType && primType == PrimType.String &&
- encodingInfo.knownEncodingIsFixedWidth => {
- //
- // Important case to optimize
- // If we can convert to a number of bits, then we should do so
- //
- val nBits = encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
- new SpecifiedLengthImplicit(this, body, nBits)
+ // we need this to evaluate before we wrap in specified length parser,
+ // so it can do any internal checks for example blobValue's check for
+ // non-explicit lengthKind
+ val body = bodyArg
+ if (
+ isSimpleType && impliedRepresentation == Representation.Text ||
+ isSimpleType && isNillable ||
+ isComplexType ||
+ lengthKind == LengthKind.Prefixed ||
+ isSimpleType && primType == PrimType.HexBinary && lengthKind == LengthKind.Pattern
+ ) {
+ lazy val bitsMultiplier = lengthUnits match {
+ case LengthUnits.Bits => 1
+ case LengthUnits.Bytes => 8
+ case LengthUnits.Characters if knownEncodingIsFixedWidth =>
+ this.knownEncodingWidthInBits
+ case _ => 0 // zero means can't multiply to get width in bits.
}
- case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
- new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
- case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
- new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
- case LengthKind.Implicit if isComplexType =>
- body // for complex types, implicit means "roll up from the bottom"
- case LengthKind.EndOfParent if isComplexType =>
- notYetImplemented("lengthKind='endOfParent' for complex type")
- case LengthKind.EndOfParent =>
- notYetImplemented("lengthKind='endOfParent' for simple type")
- case _ => {
- // TODO: implement other specified length like end of parent
- // for now, no restriction
- body
+ val lk = lengthKind
+ lk match {
+ case LengthKind.Delimited => body
Review Comment:
lengthKind=Delimted just uses body--thoughts on adding a `|| lengthKind == LengthKind.Delimited` to the condition above, making it more clear that all delimited processors do the length calculations themselves and don't need a specified length processor?
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1322,60 +1328,75 @@ trait ElementBaseGrammarMixin
* as well, by not enclosing the body in a specified length enforcer.
*/
private def specifiedLength(bodyArg: => Gram) = {
- lazy val body = bodyArg
- lazy val bitsMultiplier = lengthUnits match {
- case LengthUnits.Bits => 1
- case LengthUnits.Bytes => 8
- case LengthUnits.Characters if knownEncodingIsFixedWidth => this.knownEncodingWidthInBits
- case _ => 0 // zero means can't multiply to get width in bits.
- }
- val lk = lengthKind
- lk match {
- case LengthKind.Delimited => body
- case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
- case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
- case LengthKind.Explicit => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthExplicitCharacters(this, body)
- }
- case LengthKind.Prefixed if (bitsMultiplier != 0) =>
- new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
- case LengthKind.Prefixed => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthPrefixedCharacters(this, body)
- }
- case LengthKind.Implicit
- if isSimpleType && primType == PrimType.String &&
- encodingInfo.knownEncodingIsFixedWidth => {
- //
- // Important case to optimize
- // If we can convert to a number of bits, then we should do so
- //
- val nBits = encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
- new SpecifiedLengthImplicit(this, body, nBits)
+ // we need this to evaluate before we wrap in specified length parser,
+ // so it can do any internal checks for example blobValue's check for
+ // non-explicit lengthKind
+ val body = bodyArg
+ if (
+ isSimpleType && impliedRepresentation == Representation.Text ||
+ isSimpleType && isNillable ||
+ isComplexType ||
+ lengthKind == LengthKind.Prefixed ||
+ isSimpleType && primType == PrimType.HexBinary && lengthKind == LengthKind.Pattern
+ ) {
+ lazy val bitsMultiplier = lengthUnits match {
+ case LengthUnits.Bits => 1
+ case LengthUnits.Bytes => 8
+ case LengthUnits.Characters if knownEncodingIsFixedWidth =>
+ this.knownEncodingWidthInBits
+ case _ => 0 // zero means can't multiply to get width in bits.
}
- case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
- new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
- case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
- new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
- case LengthKind.Implicit if isComplexType =>
- body // for complex types, implicit means "roll up from the bottom"
- case LengthKind.EndOfParent if isComplexType =>
- notYetImplemented("lengthKind='endOfParent' for complex type")
- case LengthKind.EndOfParent =>
- notYetImplemented("lengthKind='endOfParent' for simple type")
- case _ => {
- // TODO: implement other specified length like end of parent
- // for now, no restriction
- body
+ val lk = lengthKind
+ lk match {
+ case LengthKind.Delimited => body
+ case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
+ case LengthKind.Explicit if bitsMultiplier != 0 =>
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ case LengthKind.Explicit => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthExplicitCharacters(this, body)
+ }
+ case LengthKind.Prefixed if (bitsMultiplier != 0) =>
+ new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
+ case LengthKind.Prefixed => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthPrefixedCharacters(this, body)
+ }
+ case LengthKind.Implicit
+ if isSimpleType && primType == PrimType.String &&
+ encodingInfo.knownEncodingIsFixedWidth => {
+ //
+ // Important case to optimize
+ // If we can convert to a number of bits, then we should do so
+ //
+ val nBits =
+ encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
+ new SpecifiedLengthImplicit(this, body, nBits)
+ }
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
+ new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
+
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
+ new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
+ case LengthKind.Implicit
+ if isSimpleType && impliedRepresentation == Representation.Binary =>
+ new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
+ case LengthKind.Implicit if isComplexType =>
+ body // for complex types, implicit means "roll up from the bottom"
+ case LengthKind.EndOfParent if isComplexType =>
+ notYetImplemented("lengthKind='endOfParent' for complex type")
+ case LengthKind.EndOfParent =>
+ notYetImplemented("lengthKind='endOfParent' for simple type")
+ case _ => {
Review Comment:
This default case isn't covered, which is probably a good thing--it probably means the condition at the top is catching all cases that really don't need a specified length parser and we aren't just accidentally falling into this case. I would suggest we either remove this case, or if Scala complains about a non-exhaustive match change it an an `Assert.impossible` with codecov disabled.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1322,60 +1328,75 @@ trait ElementBaseGrammarMixin
* as well, by not enclosing the body in a specified length enforcer.
*/
private def specifiedLength(bodyArg: => Gram) = {
- lazy val body = bodyArg
- lazy val bitsMultiplier = lengthUnits match {
- case LengthUnits.Bits => 1
- case LengthUnits.Bytes => 8
- case LengthUnits.Characters if knownEncodingIsFixedWidth => this.knownEncodingWidthInBits
- case _ => 0 // zero means can't multiply to get width in bits.
- }
- val lk = lengthKind
- lk match {
- case LengthKind.Delimited => body
- case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
- case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
- case LengthKind.Explicit => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthExplicitCharacters(this, body)
- }
- case LengthKind.Prefixed if (bitsMultiplier != 0) =>
- new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
- case LengthKind.Prefixed => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthPrefixedCharacters(this, body)
- }
- case LengthKind.Implicit
- if isSimpleType && primType == PrimType.String &&
- encodingInfo.knownEncodingIsFixedWidth => {
- //
- // Important case to optimize
- // If we can convert to a number of bits, then we should do so
- //
- val nBits = encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
- new SpecifiedLengthImplicit(this, body, nBits)
+ // we need this to evaluate before we wrap in specified length parser,
+ // so it can do any internal checks for example blobValue's check for
+ // non-explicit lengthKind
+ val body = bodyArg
+ if (
+ isSimpleType && impliedRepresentation == Representation.Text ||
+ isSimpleType && isNillable ||
+ isComplexType ||
+ lengthKind == LengthKind.Prefixed ||
+ isSimpleType && primType == PrimType.HexBinary && lengthKind == LengthKind.Pattern
+ ) {
+ lazy val bitsMultiplier = lengthUnits match {
+ case LengthUnits.Bits => 1
+ case LengthUnits.Bytes => 8
+ case LengthUnits.Characters if knownEncodingIsFixedWidth =>
+ this.knownEncodingWidthInBits
+ case _ => 0 // zero means can't multiply to get width in bits.
}
- case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
- new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
- case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
- new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
- case LengthKind.Implicit if isComplexType =>
- body // for complex types, implicit means "roll up from the bottom"
- case LengthKind.EndOfParent if isComplexType =>
- notYetImplemented("lengthKind='endOfParent' for complex type")
- case LengthKind.EndOfParent =>
- notYetImplemented("lengthKind='endOfParent' for simple type")
- case _ => {
- // TODO: implement other specified length like end of parent
- // for now, no restriction
- body
+ val lk = lengthKind
+ lk match {
+ case LengthKind.Delimited => body
+ case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
+ case LengthKind.Explicit if bitsMultiplier != 0 =>
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ case LengthKind.Explicit => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthExplicitCharacters(this, body)
+ }
+ case LengthKind.Prefixed if (bitsMultiplier != 0) =>
+ new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
+ case LengthKind.Prefixed => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthPrefixedCharacters(this, body)
+ }
+ case LengthKind.Implicit
+ if isSimpleType && primType == PrimType.String &&
+ encodingInfo.knownEncodingIsFixedWidth => {
+ //
+ // Important case to optimize
+ // If we can convert to a number of bits, then we should do so
+ //
+ val nBits =
+ encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
+ new SpecifiedLengthImplicit(this, body, nBits)
+ }
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
+ new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
+
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
+ new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
+ case LengthKind.Implicit
+ if isSimpleType && impliedRepresentation == Representation.Binary =>
+ new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
+ case LengthKind.Implicit if isComplexType =>
+ body // for complex types, implicit means "roll up from the bottom"
Review Comment:
Should we change the condition at the top to `isComplex && lengthKind != lengthKind.Implicit` and remove this case? I'm just wondering if there's value in making it so non of these cases have `body` as the result if that makes things more clear exactly when we need a specified length parser?
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/primitives/PrimitivesLengthKind.scala:
##########
@@ -186,21 +185,11 @@ case class HexBinaryEndOfBitLimit(e: ElementBase) extends Terminal(e, true) {
case class HexBinaryLengthPrefixed(e: ElementBase) extends Terminal(e, true) {
Review Comment:
I think this comment still applies to the new changes? I think it's fine to keep the `lazy val hexBinaryLengthPrefixed` grammar, but we can probably remove class and replace its uses with the `HexBinaryEndOfBitLimit` class.
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/BinaryNumberTraits.scala:
##########
@@ -165,3 +165,21 @@ trait PrefixedLengthParserMixin {
}
}
}
+
+/**
+ * This mixin doesn't require parsing the prefix length element and just uses
+ * the state's bitLimit and position to get the bitLength instead
Review Comment:
This comment is a little confusing out of context, since it isn't really clear why prefix length elements are mentioned since this doesn't really seem to be prefixed length related. This could also in theory be used on many of the parsers that rely on bit limit for length, which isn't specific to prefixed length. Maybe instead say something like
> Some parsers do not calculate their own length, but instead expect another parser to set the bit limit and then they use that bit limit as the length. An example of this is prefix length parsers. This trait can be used by those parsers to do determine the length based on the bitLimit and position.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1322,60 +1328,75 @@ trait ElementBaseGrammarMixin
* as well, by not enclosing the body in a specified length enforcer.
*/
private def specifiedLength(bodyArg: => Gram) = {
- lazy val body = bodyArg
- lazy val bitsMultiplier = lengthUnits match {
- case LengthUnits.Bits => 1
- case LengthUnits.Bytes => 8
- case LengthUnits.Characters if knownEncodingIsFixedWidth => this.knownEncodingWidthInBits
- case _ => 0 // zero means can't multiply to get width in bits.
- }
- val lk = lengthKind
- lk match {
- case LengthKind.Delimited => body
- case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
- case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
- case LengthKind.Explicit => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthExplicitCharacters(this, body)
- }
- case LengthKind.Prefixed if (bitsMultiplier != 0) =>
- new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
- case LengthKind.Prefixed => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthPrefixedCharacters(this, body)
- }
- case LengthKind.Implicit
- if isSimpleType && primType == PrimType.String &&
- encodingInfo.knownEncodingIsFixedWidth => {
- //
- // Important case to optimize
- // If we can convert to a number of bits, then we should do so
- //
- val nBits = encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
- new SpecifiedLengthImplicit(this, body, nBits)
+ // we need this to evaluate before we wrap in specified length parser,
+ // so it can do any internal checks for example blobValue's check for
+ // non-explicit lengthKind
+ val body = bodyArg
+ if (
+ isSimpleType && impliedRepresentation == Representation.Text ||
+ isSimpleType && isNillable ||
+ isComplexType ||
+ lengthKind == LengthKind.Prefixed ||
+ isSimpleType && primType == PrimType.HexBinary && lengthKind == LengthKind.Pattern
+ ) {
+ lazy val bitsMultiplier = lengthUnits match {
+ case LengthUnits.Bits => 1
+ case LengthUnits.Bytes => 8
+ case LengthUnits.Characters if knownEncodingIsFixedWidth =>
+ this.knownEncodingWidthInBits
+ case _ => 0 // zero means can't multiply to get width in bits.
}
- case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
- new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
- case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
- new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
- case LengthKind.Implicit if isComplexType =>
- body // for complex types, implicit means "roll up from the bottom"
- case LengthKind.EndOfParent if isComplexType =>
- notYetImplemented("lengthKind='endOfParent' for complex type")
- case LengthKind.EndOfParent =>
- notYetImplemented("lengthKind='endOfParent' for simple type")
- case _ => {
- // TODO: implement other specified length like end of parent
- // for now, no restriction
- body
+ val lk = lengthKind
+ lk match {
+ case LengthKind.Delimited => body
+ case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
+ case LengthKind.Explicit if bitsMultiplier != 0 =>
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ case LengthKind.Explicit => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthExplicitCharacters(this, body)
+ }
+ case LengthKind.Prefixed if (bitsMultiplier != 0) =>
+ new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
+ case LengthKind.Prefixed => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthPrefixedCharacters(this, body)
+ }
+ case LengthKind.Implicit
+ if isSimpleType && primType == PrimType.String &&
+ encodingInfo.knownEncodingIsFixedWidth => {
+ //
+ // Important case to optimize
+ // If we can convert to a number of bits, then we should do so
+ //
+ val nBits =
+ encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
+ new SpecifiedLengthImplicit(this, body, nBits)
+ }
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
+ new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
+
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
+ new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
Review Comment:
This case has no coverage, which I think makes sense since only prefixed length hexBinary uses a specified length parser. All other hexBinary parsers calculate their own lengths. Should we remove this case to avoid dead code and possible confusion with how hexBinary is implemented?
##########
daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/unparsers/runtime1/SpecifiedLengthUnparsers.scala:
##########
@@ -200,7 +124,7 @@ trait CalculatedPrefixedLengthUnparserMixin {
UnparseError(
One(state.schemaFileLocation),
One(state.currentLocation),
- s"The calculated value of ${elem.namedQName} ($adjustedLenInUnits) failed check due to ${check.errMsg}"
+ s"The calculated value of ${plElem.namedQName} ($adjustedLenInUnits) failed check due to ${check.errMsg}"
Review Comment:
Is switching to `plElem` correct? Although the check is done against `plElem`, `plElem` is really an ephemeral quasi-element that doesn't exist in the real infoset, so it might be confusing to mention it in an error message. Maybe this instead wants to be something like "The calculated prefix length of ${elem.namedQName} ..."?
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/ElementBaseRuntime1Mixin.scala:
##########
@@ -65,7 +65,8 @@ trait ElementBaseRuntime1Mixin { self: ElementBase =>
// no reason (unless it is referenced in a contentLength expression).
val mightHaveSuspensions = (maybeFixedLengthInBits.isDefined && couldHaveSuspensions)
- isReferenced || mightHaveSuspensions
+ // we want to capture contentlength when LK = prefixed
Review Comment:
I think we should explain why we need to capture content length. Maybe something like
> Some prefixed length unparsers are unable to calculate the prefixed length of the field. Instead, they unparse the field to a buffer and the captured content length of the buffer is used. For this reason, prefixed length elements must capture content length for unparse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1909594142
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1341,7 +1356,13 @@ trait ElementBaseGrammarMixin
case LengthKind.Delimited => body
case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ if (isSimpleType && primType == PrimType.HexBinary) {
+ // hexBinary has some checks that need to be done that SpecifiedLengthExplicit
+ // gets in the way of
Review Comment:
I would not refactor this now. This code has gotten kind of undisciplined and been that way for a long time now.
Our performance last we checked was in the ballpark of 2x what a programmer would write by hand for simple binary data.
If we start reorganizing this sort of thing we need to rerun those tests. We're too close to a release for that.
Fixing these bugs does seem worth it however.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1387,14 +1381,15 @@ trait ElementBaseGrammarMixin
}
case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
+ if isSimpleType &&
+ impliedRepresentation == Representation.Binary &&
+ primType != PrimType.HexBinary =>
new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
Review Comment:
I suggest parking this cleanup as a separate ticket. Unnecessary parsers are not optimized out in many cases I bet, and this is just one such case where specified length stuff is unnecessary but not removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#issuecomment-2582944920
Created DAFFODIL-2967 to ensure we're properly adding/not adding the schemas
@mbeckerle , the piece of code you highlighted as being better off removed, removing that bit caused test failures iirc!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#issuecomment-2635074861
ChatGPT thinks that yes there are issues with CodeCov and Scala code:
https://chatgpt.com/share/67a28042-0944-800f-b2e5-95c1856c91d6
On Tue, Feb 4, 2025 at 1:55 PM olabusayoT ***@***.***> wrote:
> ***@***.**** commented on this pull request.
> ------------------------------
>
> In
> daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala
> <https://github.com/apache/daffodil/pull/1338#discussion_r1941736405>:
>
> > + }
> + case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
> + new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
> +
> + case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
> + new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
> + case LengthKind.Implicit
> + if isSimpleType && impliedRepresentation == Representation.Binary =>
> + new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
> + case LengthKind.Implicit if isComplexType =>
> + body // for complex types, implicit means "roll up from the bottom"
> + case LengthKind.EndOfParent if isComplexType =>
> + notYetImplemented("lengthKind='endOfParent' for complex type")
> + case LengthKind.EndOfParent =>
> + notYetImplemented("lengthKind='endOfParent' for simple type")
> + case _ => {
>
> Maybe there's a bug with code coverage, hexBinary_bits_be_msbf definitely
> hits the default case
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/daffodil/pull/1338#discussion_r1941736405>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AALUDAZJDC6SSICL3X2HUDD2OEEKLAVCNFSM6AAAAABQADPAVKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKOJTG43TOMBUHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1808993978
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/SpecifiedLengthParsers.scala:
##########
@@ -85,8 +85,8 @@ sealed abstract class SpecifiedLengthParserBase(eParser: Parser, erd: RuntimeDat
if (pState.processorStatus ne Success) return
val finalEndPos0b = startingBitPos0b + nBits
- // if we haven't already set the value length, set it now
- if (pState.infoset.valueLength.isEndUndef)
+ // we want to capture the length before we do any skipping
Review Comment:
Might be worth adding a comment that value length of simple types is captured by the eParser if needed.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/ElementBaseRuntime1Mixin.scala:
##########
@@ -89,17 +89,19 @@ trait ElementBaseRuntime1Mixin { self: ElementBase =>
//
// For complex elements with specified length, value length is captured in
// the specified length parsers, since they handle skipping unused
- // element regions. For complex elements, this means lengthKind is not
+ // element regions, and the length capturing has to be done before that.
+ // For complex elements, this means lengthKind is not
// implicit or delimited.
//
// So for these cases we do not want to capture value length with the
// Capture{Start,End}OfValueLengthParsers, since those lengths are captured
// by the value parsers
- val capturedByParsers =
- (isSimpleType && (impliedRepresentation == Representation.Text || lengthKind == LengthKind.Delimited)) ||
+ val capturedByValueParsers =
+ (isSimpleType && (
+ primType == PrimType.String || lengthKind == LengthKind.Delimited)) ||
Review Comment:
I'm wondering if this change is correct?
For example, say we have this element:
```xml
<xs:element name="foo" type="xs:int" dfdl:representation="text" dfdl:trimKind="padChar" dfdl:lengthKind="explicit" dfdl:length="10" ... />
```
So a fixed length text integer with padding. In this case I think what Daffodil does is it create a String parser to parse the fixed length string and remove padding, and then creates another parser to convert that string to an actual integer.
So in that case, even though the primType is not String, I think the String parser will still be used to capture the value length after padding is removed. So I *think* `impliedRepresentation == Text` is still needed?
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/runtime1/ElementBaseRuntime1Mixin.scala:
##########
@@ -24,9 +24,9 @@ import org.apache.daffodil.core.dsom.PrimitiveType
import org.apache.daffodil.core.dsom.Root
import org.apache.daffodil.core.dsom.SimpleTypeDefBase
Review Comment:
I wonder if we should modify `setAbsStartBitPos0bInBits` so it does `Assert.invariant(maybeStartPos0bInBits.isEmpty)` and something similar for `setAbsEndBitPos0bInBits`? I think those assertion would have caught this bug since if it fails it means we have set the absolute bit position twice, which we should never do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1910498123
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1341,7 +1356,13 @@ trait ElementBaseGrammarMixin
case LengthKind.Delimited => body
case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ if (isSimpleType && primType == PrimType.HexBinary) {
+ // hexBinary has some checks that need to be done that SpecifiedLengthExplicit
+ // gets in the way of
+ body
+ } else {
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ }
case LengthKind.Explicit => {
Assert.invariant(!knownEncodingIsFixedWidth)
Assert.invariant(lengthUnits eq LengthUnits.Characters)
Review Comment:
I'm not sure this is a cleanup of existing issues. I think this change could possibly lead to adding duplicate parsers and might slow things down. So it's not that we already have duplicates, but this change could lead to new duplicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1827928917
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1341,7 +1356,13 @@ trait ElementBaseGrammarMixin
case LengthKind.Delimited => body
case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ if (isSimpleType && primType == PrimType.HexBinary) {
+ // hexBinary has some checks that need to be done that SpecifiedLengthExplicit
+ // gets in the way of
Review Comment:
I think instead of this comment, we can say HexBinary has it's own HexBinarySpecifiedLength parser that handles calculating the length, so we do not need the SpecifiedLengthExplicit parser?
In fact, do we need to exclude a number of other primitive types that do their own explicit length handling? Looking at the current code base, I think maybe only simple types that are strings and complex types use the SpecifiedLengthExplicit parser? I think all other primitives implement their own specified length handling?
So maybe this wants to be
```scala
if (isComplexType || primType == PrimType.String) {
SpecifiedLengthExplicit(...)
} else {
// non-string simple types have their own custom parsers/unparsers for handling explicit lengths
body
}
```
In fact, I wonder if we eventually want to refactor all of this to completely get rid of all the custom explicit/implicit length parsers? We just have various SpecifiedLength parser that sets a bit limit (based on a pattern, a prefix length, evaluaating a length expression etc) and then we just have a single parser that just reads all bit up until that current bit limit. Separation of concerns kind of thing. It would get rid of this condiation and all these BinaryIntegerKnownLength/RuntimeLength/PrefixLength/etc parsers. There's just a single BinaryNumberParser, and it just gets the length from the bitLimit.
Maybe that generality would take performance hit? I'm also not exactly sure how that would work with unparsing--the SpecifiedLengthUnparser would need to somehow pass the calculated length to the child unparser, I guess it could still use bitLimit since that is a thing in UState?
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/primitives/PrimitivesLengthKind.scala:
##########
@@ -186,21 +185,11 @@ case class HexBinaryEndOfBitLimit(e: ElementBase) extends Terminal(e, true) {
case class HexBinaryLengthPrefixed(e: ElementBase) extends Terminal(e, true) {
Review Comment:
This class is now exactly the same as HexBinaryEndOfBitLimit, suggest we just delete it an use that for prefixed hex binary.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/primitives/PrimitivesBCD.scala:
##########
@@ -52,20 +52,10 @@ class BCDIntegerKnownLength(val e: ElementBase, lengthInBits: Long) extends Term
class BCDIntegerPrefixedLength(val e: ElementBase) extends Terminal(e, true) {
- override lazy val parser = new BCDIntegerPrefixedLengthParser(
- e.elementRuntimeData,
- e.prefixedLengthBody.parser,
- e.prefixedLengthElementDecl.elementRuntimeData,
- e.lengthUnits,
- e.prefixedLengthAdjustmentInUnits
- )
+ override lazy val parser = new BCDIntegerPrefixedLengthParser(e.elementRuntimeData)
Review Comment:
I wonder if we want to rename these something like `BCDIntegerBitLimitLengthParser` and `BCDIntegerMinimumLengthUnparser`, making it clear that they aren't really doing anything specifically with prefixed length, and better describes the behavior of how they actuall parse/unparse things?
And when we implement things like lengthKind endOfParent, we could probably just use the same BitLimitParser, for example.
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/BinaryNumberTraits.scala:
##########
@@ -165,3 +165,21 @@ trait PrefixedLengthParserMixin {
}
}
}
+
+/**
+ * This mixin doesn't require parsing the prefix length element and just uses
+ * the state's bitLimit and position to get the bitLength instead
+ */
+trait PrefixedLengthParserMixin2 {
Review Comment:
Need a different name for this. Numbers are not descriptive enough. Maybe we renames these something like `PrefixLengthFromParserMixin` and `PrefixLengthFromBitLimitMixin`? Something to make it more clear how they differ without having to read the code.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1341,7 +1356,13 @@ trait ElementBaseGrammarMixin
case LengthKind.Delimited => body
case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ if (isSimpleType && primType == PrimType.HexBinary) {
+ // hexBinary has some checks that need to be done that SpecifiedLengthExplicit
+ // gets in the way of
+ body
+ } else {
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ }
case LengthKind.Explicit => {
Assert.invariant(!knownEncodingIsFixedWidth)
Assert.invariant(lengthUnits eq LengthUnits.Characters)
Review Comment:
Below this we have cases for implicit lengths. Do we need to do anything special for non-string simple types? I think those primitives have custom parsers that handle the implict length logic and don't need a SpecifiedLengthImplicit gramar? My concern is we could be adding that grammar and it would do something like set a bit limit, but the child paser that actually parsrers a the thing would just use it's own calculate and wouldn't need the bit limit, so we are just wasting effort.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -72,13 +72,21 @@ trait ElementBaseGrammarMixin
}
}
- protected lazy val isDelimitedPrefixedPattern = {
+ protected lazy val isPrefixed: Boolean = {
+ import LengthKind._
+ lengthKind match {
+ case Prefixed => true
+ case _ => false
+ }
+ }
Review Comment:
This can just be `lengthKind eq LengthKind.Prefixed`. Don't really need a match/case if we are just going to return true/false.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -72,13 +72,21 @@ trait ElementBaseGrammarMixin
}
}
- protected lazy val isDelimitedPrefixedPattern = {
+ protected lazy val isPrefixed: Boolean = {
+ import LengthKind._
+ lengthKind match {
+ case Prefixed => true
+ case _ => false
+ }
+ }
+
+ protected lazy val isDelimitedPrefixedPattern: Boolean = {
import LengthKind._
lengthKind match {
case Delimited =>
true // don't test for hasDelimiters because it might not be our delimiter, but a surrounding group's separator, or it's terminator, etc.
case Pattern => true
- case Prefixed => true
+ case Prefixed => isPrefixed
Review Comment:
I would suggest just returning true here, isPrefixed doesn't do anything except check lengthKind == Prefixed, which is exactly what this match case does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1803438203
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/SpecifiedLengthParsers.scala:
##########
@@ -85,7 +85,9 @@ sealed abstract class SpecifiedLengthParserBase(eParser: Parser, erd: RuntimeDat
if (pState.processorStatus ne Success) return
val finalEndPos0b = startingBitPos0b + nBits
- captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
+ // if we haven't already set the value length, set it now
+ if (pState.infoset.valueLength.isEndUndef)
+ captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
Review Comment:
> Is the issue here that some `eParsers` do not capture their own length and expect it to be captured here, but other parsers do capture value length (because they handle things like trimming), so sometimes there are two parsers trying to capture length for the same thing?
That's right. For example, SpecifiedLengthExplicitParser doesn't capture its own length.
> That makes me wonder if we shouldn't be capturing value length here at all, and instead we should be using the CaptureValueLengthStart/End parsers for those parsers that don't capture value length?
I'll look into implementing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1840490014
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1387,14 +1381,15 @@ trait ElementBaseGrammarMixin
}
case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
+ if isSimpleType &&
+ impliedRepresentation == Representation.Binary &&
+ primType != PrimType.HexBinary =>
new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
Review Comment:
This condition feels like it doesn't exclude enough things. For example, this matches all binary types except hex binary. But, for example, integers with length kind implicit use something like `BinaryIntegerKnownLength` primitive/parser, which doesn't rely on SpecifiedLengthImplicit parser. It doesn't necessarily hurt, but it's just unnecessary work and might slow things down, since the BinaryIntegerKnownLengthParser is going to do that work.
I'm wonder if it's just string types that really make use of specified length stuff?
##########
daffodil-codegen-c/src/main/scala/org/apache/daffodil/codegen/c/DaffodilCCodeGenerator.scala:
##########
@@ -289,7 +289,8 @@ object DaffodilCCodeGenerator
case g: ElementParseAndUnspecifiedLength =>
elementParseAndUnspecifiedLengthGenerateCode(g, cgState)
case g: ElementUnused => noop(g)
- case g: HexBinaryLengthPrefixed => hexBinaryLengthPrefixedGenerateCode(g.e, cgState)
+ case g: HexBinaryEndOfBitLimit if g.e.isPrefixed =>
Review Comment:
Hmm, maybe my suggestion was wrong and the primitive still wants to be something like HexBInaryLengthPrefixed so that the grammar is obvious and things like code generators/etc can use the obvious grammar names? The parsers generated don't necessarily have to match the names.
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1387,14 +1381,15 @@ trait ElementBaseGrammarMixin
}
case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
+ if isSimpleType &&
+ impliedRepresentation == Representation.Binary &&
+ primType != PrimType.HexBinary =>
new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
- case LengthKind.Implicit if isComplexType =>
+ case LengthKind.Implicit =>
body // for complex types, implicit means "roll up from the bottom"
+ // for simple types, the primitives have custom parsers that handle implicit length logic
+ // and don't use the limit provided by the SpecifiedLengthImplicit parser
Review Comment:
Can you move this comment above `body`? Single line comments can be on the same line, but multi-line comments should start before the line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1909607293
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1341,7 +1356,13 @@ trait ElementBaseGrammarMixin
case LengthKind.Delimited => body
case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ if (isSimpleType && primType == PrimType.HexBinary) {
+ // hexBinary has some checks that need to be done that SpecifiedLengthExplicit
+ // gets in the way of
+ body
+ } else {
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ }
case LengthKind.Explicit => {
Assert.invariant(!knownEncodingIsFixedWidth)
Assert.invariant(lengthUnits eq LengthUnits.Characters)
Review Comment:
Suggest create a cleanup issue ticket for this. It's a performance (and maintainability) improvement but not specifically about this PR's primary goal is it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1941736405
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1322,60 +1328,75 @@ trait ElementBaseGrammarMixin
* as well, by not enclosing the body in a specified length enforcer.
*/
private def specifiedLength(bodyArg: => Gram) = {
- lazy val body = bodyArg
- lazy val bitsMultiplier = lengthUnits match {
- case LengthUnits.Bits => 1
- case LengthUnits.Bytes => 8
- case LengthUnits.Characters if knownEncodingIsFixedWidth => this.knownEncodingWidthInBits
- case _ => 0 // zero means can't multiply to get width in bits.
- }
- val lk = lengthKind
- lk match {
- case LengthKind.Delimited => body
- case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
- case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
- case LengthKind.Explicit => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthExplicitCharacters(this, body)
- }
- case LengthKind.Prefixed if (bitsMultiplier != 0) =>
- new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
- case LengthKind.Prefixed => {
- Assert.invariant(!knownEncodingIsFixedWidth)
- Assert.invariant(lengthUnits eq LengthUnits.Characters)
- new SpecifiedLengthPrefixedCharacters(this, body)
- }
- case LengthKind.Implicit
- if isSimpleType && primType == PrimType.String &&
- encodingInfo.knownEncodingIsFixedWidth => {
- //
- // Important case to optimize
- // If we can convert to a number of bits, then we should do so
- //
- val nBits = encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
- new SpecifiedLengthImplicit(this, body, nBits)
+ // we need this to evaluate before we wrap in specified length parser,
+ // so it can do any internal checks for example blobValue's check for
+ // non-explicit lengthKind
+ val body = bodyArg
+ if (
+ isSimpleType && impliedRepresentation == Representation.Text ||
+ isSimpleType && isNillable ||
+ isComplexType ||
+ lengthKind == LengthKind.Prefixed ||
+ isSimpleType && primType == PrimType.HexBinary && lengthKind == LengthKind.Pattern
+ ) {
+ lazy val bitsMultiplier = lengthUnits match {
+ case LengthUnits.Bits => 1
+ case LengthUnits.Bytes => 8
+ case LengthUnits.Characters if knownEncodingIsFixedWidth =>
+ this.knownEncodingWidthInBits
+ case _ => 0 // zero means can't multiply to get width in bits.
}
- case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
- new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
- case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
- new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
- case LengthKind.Implicit if isComplexType =>
- body // for complex types, implicit means "roll up from the bottom"
- case LengthKind.EndOfParent if isComplexType =>
- notYetImplemented("lengthKind='endOfParent' for complex type")
- case LengthKind.EndOfParent =>
- notYetImplemented("lengthKind='endOfParent' for simple type")
- case _ => {
- // TODO: implement other specified length like end of parent
- // for now, no restriction
- body
+ val lk = lengthKind
+ lk match {
+ case LengthKind.Delimited => body
+ case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
+ case LengthKind.Explicit if bitsMultiplier != 0 =>
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ case LengthKind.Explicit => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthExplicitCharacters(this, body)
+ }
+ case LengthKind.Prefixed if (bitsMultiplier != 0) =>
+ new SpecifiedLengthPrefixed(this, body, bitsMultiplier)
+ case LengthKind.Prefixed => {
+ Assert.invariant(!knownEncodingIsFixedWidth)
+ Assert.invariant(lengthUnits eq LengthUnits.Characters)
+ new SpecifiedLengthPrefixedCharacters(this, body)
+ }
+ case LengthKind.Implicit
+ if isSimpleType && primType == PrimType.String &&
+ encodingInfo.knownEncodingIsFixedWidth => {
+ //
+ // Important case to optimize
+ // If we can convert to a number of bits, then we should do so
+ //
+ val nBits =
+ encodingInfo.knownFixedWidthEncodingInCharsToBits(this.maxLength.longValue)
+ new SpecifiedLengthImplicit(this, body, nBits)
+ }
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
+ new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
+
+ case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
+ new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
+ case LengthKind.Implicit
+ if isSimpleType && impliedRepresentation == Representation.Binary =>
+ new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
+ case LengthKind.Implicit if isComplexType =>
+ body // for complex types, implicit means "roll up from the bottom"
+ case LengthKind.EndOfParent if isComplexType =>
+ notYetImplemented("lengthKind='endOfParent' for complex type")
+ case LengthKind.EndOfParent =>
+ notYetImplemented("lengthKind='endOfParent' for simple type")
+ case _ => {
Review Comment:
Maybe there's a bug with code coverage, hexBinary_bits_be_msbf definitely hits the default case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "mbeckerle (via GitHub)" <gi...@apache.org>.
mbeckerle commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1807082323
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/SpecifiedLengthParsers.scala:
##########
@@ -85,7 +85,9 @@ sealed abstract class SpecifiedLengthParserBase(eParser: Parser, erd: RuntimeDat
if (pState.processorStatus ne Success) return
val finalEndPos0b = startingBitPos0b + nBits
- captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
+ // if we haven't already set the value length, set it now
+ if (pState.infoset.valueLength.isEndUndef)
+ captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
Review Comment:
All the complexity around whether to insert a value-length capture parser at the start/end is about symmetry of the parser with the way the unparser works.
Capturing length when parsing is easy. Record start position when parser is entered, ending position when it is exited. Probably so light weight that it's not worth trying to optimize it away.
Unparsing... not so much. The unparser has to unwind and allow unparsing to continue while the value length is suspended. Definitely we want none of this overhead for elements which no expression references their length. We only want capture of length overheads when it is essential. I am also hoping that some of this is simpler for prefixed lengths, which don't work for unparsing with suspensions, etc. currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1803544680
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/SpecifiedLengthParsers.scala:
##########
@@ -85,7 +85,9 @@ sealed abstract class SpecifiedLengthParserBase(eParser: Parser, erd: RuntimeDat
if (pState.processorStatus ne Success) return
val finalEndPos0b = startingBitPos0b + nBits
- captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
+ // if we haven't already set the value length, set it now
+ if (pState.infoset.valueLength.isEndUndef)
+ captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
Review Comment:
The `isReferenced` variable is used to avoid capturing value lengths that will never be used. For example, if we have something like this:
```xml
<element name="a" type="xs:int" ... />
```
If nothing ever references the value length of `a, then we shouldn't insert `CaptureValueLengthStart/End` parsers, regardless of the value of `capturedByParsers`. But if something does reference the length, e.g.
```xml
<element name="len" type="xs:int" dfdl:inputValueCalc="{ dfdl:valueLength(../a, 'bytes')) } />
```
The we do need to capture the value length of `a`. So `isReferenced` is needed to disabled capturing when it's not needed. I don't think we can get rid of it.
Usually we capture the length via the `CaptureValueLengthStart/End` parsers, but sometimes those parsers aren't smart enough so it must be done by a separate parser that has enough information to capture length correctly (e.g. StringOfSpecifiedLength that knows about pad/trim stuff and how to exclude it from the value length). The `capturedByParsers` val is supposed to figure out which are those special cases so we don't unnecessarily insert `CaptureValueLengthStart/End` parsers for parsers that do it themselves.
It seems like SpecifiedLengthParserBase doesn't have enough information to know how to correctly capture value length (since sometimes the subparser is the thing that knows how to do it). So that probably means it shouldn't be capturing the length at all. But that then means other parsers that relied on that behavior either need to call `captureValueLength` themselves, or `capturedByParsers` needs to be false in some cases so that we insert the `CaptureValueLengthStart/End` parsers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1803502665
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/SpecifiedLengthParsers.scala:
##########
@@ -85,7 +85,9 @@ sealed abstract class SpecifiedLengthParserBase(eParser: Parser, erd: RuntimeDat
if (pState.processorStatus ne Success) return
val finalEndPos0b = startingBitPos0b + nBits
- captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
+ // if we haven't already set the value length, set it now
+ if (pState.infoset.valueLength.isEndUndef)
+ captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
Review Comment:
> ElementBaseRuntime1Mixin has a capturedByParsers val that tries to list which parsers capture their own value length. Maybe that logic isn't completely correct, and the work around (that feels wrong) is we just always capture the length here?
I think it's the "isReferenced" boolean that may be the issue, if we remove that, then we'll always captureValueLength if capturedByParsers is false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1842613813
##########
daffodil-codegen-c/src/main/scala/org/apache/daffodil/codegen/c/DaffodilCCodeGenerator.scala:
##########
@@ -289,7 +289,8 @@ object DaffodilCCodeGenerator
case g: ElementParseAndUnspecifiedLength =>
elementParseAndUnspecifiedLengthGenerateCode(g, cgState)
case g: ElementUnused => noop(g)
- case g: HexBinaryLengthPrefixed => hexBinaryLengthPrefixedGenerateCode(g.e, cgState)
+ case g: HexBinaryEndOfBitLimit if g.e.isPrefixed =>
Review Comment:
Sorry that wasn't clear. I'm suggesting that we should keep the old grammar name `HexBinaryLengthPrefixed` so this change isn't needed. This way the grammar is a more clear for things like the code generator that examine it. But it's still fine to call the parsers `BitLimitLength/MinimumLength` etc from that grammar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1842641968
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1387,14 +1381,15 @@ trait ElementBaseGrammarMixin
}
case LengthKind.Implicit if isSimpleType && primType == PrimType.String =>
new SpecifiedLengthImplicitCharacters(this, body, this.maxLength.longValue)
-
- case LengthKind.Implicit if isSimpleType && primType == PrimType.HexBinary =>
- new SpecifiedLengthImplicit(this, body, this.maxLength.longValue * bitsMultiplier)
case LengthKind.Implicit
- if isSimpleType && impliedRepresentation == Representation.Binary =>
+ if isSimpleType &&
+ impliedRepresentation == Representation.Binary &&
+ primType != PrimType.HexBinary =>
new SpecifiedLengthImplicit(this, body, implicitBinaryLengthInBits)
Review Comment:
I think that means we only need the SpecifiedLengthImplicit when we are trying to parse the nillable part of a number. For example, the way things currently work is we have something like a `SimpleNilOrValueParser` and that has two children parsers, one is the nil parser (which sounds like it requires SpecifiedLengthImplicit) and the other is the binary paser (which probably does not need SpecifiedLenghtImplicit).
I wonder if captureLengthRegion needs a new paramater (e.g. `forNilContent: Boolean`) which is passed into `specifiedLength`. This way `specifiedLength` can do different things depending on if it's trying to represent nil content or non-nil content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1910503389
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1341,7 +1356,13 @@ trait ElementBaseGrammarMixin
case LengthKind.Delimited => body
case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ if (isSimpleType && primType == PrimType.HexBinary) {
+ // hexBinary has some checks that need to be done that SpecifiedLengthExplicit
+ // gets in the way of
+ body
+ } else {
+ new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ }
case LengthKind.Explicit => {
Assert.invariant(!knownEncodingIsFixedWidth)
Assert.invariant(lengthUnits eq LengthUnits.Characters)
Review Comment:
Also, I believe this bug just fixes issues with prefixed lengths, which I think has a number of other outstanding issues. So even if we fix this one, I'm not sure prefixedLength is still going to be safe to use. Feels like it's less criticial to get into this release, and due to possible regressions is worth pushing to the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT merged PR #1338:
URL: https://github.com/apache/daffodil/pull/1338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "olabusayoT (via GitHub)" <gi...@apache.org>.
olabusayoT commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1828319687
##########
daffodil-core/src/main/scala/org/apache/daffodil/core/grammar/ElementBaseGrammarMixin.scala:
##########
@@ -1341,7 +1356,13 @@ trait ElementBaseGrammarMixin
case LengthKind.Delimited => body
case LengthKind.Pattern => new SpecifiedLengthPattern(this, body)
case LengthKind.Explicit if bitsMultiplier != 0 =>
- new SpecifiedLengthExplicit(this, body, bitsMultiplier)
+ if (isSimpleType && primType == PrimType.HexBinary) {
+ // hexBinary has some checks that need to be done that SpecifiedLengthExplicit
+ // gets in the way of
Review Comment:
@mbeckerle , any thoughts on refactoring the code to have various specified length parser as described above, and any idea on how that would work with unparsing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix Bug with valueLength being overwritten after Trim [daffodil]
Posted by "stevedlawrence (via GitHub)" <gi...@apache.org>.
stevedlawrence commented on code in PR #1338:
URL: https://github.com/apache/daffodil/pull/1338#discussion_r1803084635
##########
daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/SpecifiedLengthParsers.scala:
##########
@@ -85,7 +85,9 @@ sealed abstract class SpecifiedLengthParserBase(eParser: Parser, erd: RuntimeDat
if (pState.processorStatus ne Success) return
val finalEndPos0b = startingBitPos0b + nBits
- captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
+ // if we haven't already set the value length, set it now
+ if (pState.infoset.valueLength.isEndUndef)
+ captureValueLength(pState, ULong(startingBitPos0b), ULong(dis.bitPos0b))
Review Comment:
Is the issue here that some `eParsers` do not capture their own length and expect it to be captured here, but other parsers do capture value length (because they handle things like trimming), so sometimes there are two parsers trying to capture length for the same thing?
That makes me wonder if we shouldn't be capturing value length here at all, and instead we should be using the `CaptureValueLengthStart/End` parsers for those parsers that don't capture value length?
ElementBaseRuntime1Mixin has a `capturedByParsers` val that tries to list which parsers capture their own value length. Maybe that logic isn't completely correct, and the work around (that feels wrong) is we just always capture the length here?
##########
daffodil-test/src/test/resources/org/apache/daffodil/section12/lengthKind/PrefixedTests.tdml:
##########
@@ -17,14 +17,14 @@
-->
<tdml:testSuite suiteName="PrefixedTests"
- description="Section 12 - lengthKind=prefixed" xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
- xmlns:xs="http://www.w3.org/2001/XMLSchema"
- xmlns:fn="http://www.w3.org/2005/xpath-functions"
- xmlns:ex="http://example.com"
- xmlns:tns="http://example.com"
- xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
- defaultRoundTrip="onePass">
+ description="Section 12 - lengthKind=prefixed" xmlns:tdml="http://www.ibm.com/xmlns/dfdl/testData"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+ xmlns:fn="http://www.w3.org/2005/xpath-functions"
+ xmlns:ex="http://example.com"
+ xmlns:tns="http://example.com"
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions"
+ defaultRoundTrip="onePass" xmlns:x="http://www.ogf.org/dfdl/dfdl-1.0/">
Review Comment:
Fix indentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@daffodil.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org