You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by "Adams, Joshua" <ja...@owlcyberdefense.com> on 2021/11/23 18:04:50 UTC
DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
I've been taking a look at this bug and am trying to figure out what the correct behavior for this situation would be. Currently, attempting to call dfdlx:lookAhead as part of an expression for the default value of a newVariableInstance will result in a UsageError being thrown when attempting to get the bitOrder, as the VariableRuntimeData is descended from NonTermRuntimeData.
It doesn't seem unreasonable to want to use lookAhead from inside a newVariableInstance, but I'm not sure there is an easy way to get the parent sequence/element that contains the annotation where the newVariableInstance is defined. I noticed that for elements when the expression for byteOrder isn't defined, it just defaults to BigEndian with comments stating that this likely means that the element is likely hexBinary, in which case the byte order should be ignored. I don't know if it makes sense to do something like this and just defauilt bitOrder for VariableRuntimeData's to MSBF, as from the examples of lookAhead that I've seen involve looking into a hexBinary element.
Thoughts?
Josh
Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Posted by Mike Beckerle <mb...@apache.org>.
No I think in a regular define variable there is not only no infoset when
the default expression is evaluated, but there is also no data. So it has
to be an SDE.
On Tue, Nov 23, 2021, 4:02 PM Adams, Joshua <ja...@owlcyberdefense.com>
wrote:
> Out of curiosity, would look ahead work at all if it was used as the
> default value of a regular defineVariable statement or is that not
> allowed? The spec says that the defaultValue of a defineVariable statement
> cannot reference the infoset, but using a lookAhead is a little different
> than referencing a forward path.
>
> Josh
> ________________________________
> From: Mike Beckerle <mb...@apache.org>
> Sent: Tuesday, November 23, 2021 3:27 PM
> To: dev@daffodil.apache.org <de...@daffodil.apache.org>
> Subject: Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
>
> Presumably the same issue would arise if dfdlx:lookahead was called from a
> setVariable annotation on a sequence also.
>
> I happen to know there is a requirement for this to reach into data at
> boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
> matter for this function.
>
> I claim the right thing is for the bitOrder to come from the model group on
> which the newVariableInstance or setVariable is residing.
>
> My reasoning is that if you call dfdlx:lookAhead from the choiceDispatchKey
> expression of a choice, the bitOrder is that of the choice model group.
> That's how it works today.
> There should be no difference between expressing a dfdl:choiceDispatchKey
> that directly uses a dfdlx:lookAhead call, and one that does so by way of a
> new variable instance.
>
> <xs:choice>
> <xs:annotation>
> <xs:appinfo ...>
> <dfdl:newVariableInstance ref="a:flag" defaultValue='{
> dfdlx:lookAhead(....) }'/>
> <dfdl:choice choiceDispatchKey='{ $a:flag }'/>
> </xs:appinfo>
> </xs:annotation>
> .... choice branches here
> </xs:choice>
>
> So I think the right answer is to just require the enclosing model group's
> model-group runtime data object be passed down to the newVariableInstance
> or setVariable primitive parser/unparser.
>
> On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua <ja...@owlcyberdefense.com>
> wrote:
>
> > I've been taking a look at this bug and am trying to figure out what the
> > correct behavior for this situation would be. Currently, attempting to
> > call dfdlx:lookAhead as part of an expression for the default value of a
> > newVariableInstance will result in a UsageError being thrown when
> > attempting to get the bitOrder, as the VariableRuntimeData is descended
> > from NonTermRuntimeData.
> >
> > It doesn't seem unreasonable to want to use lookAhead from inside a
> > newVariableInstance, but I'm not sure there is an easy way to get the
> > parent sequence/element that contains the annotation where the
> > newVariableInstance is defined. I noticed that for elements when the
> > expression for byteOrder isn't defined, it just defaults to BigEndian
> with
> > comments stating that this likely means that the element is likely
> > hexBinary, in which case the byte order should be ignored. I don't know
> if
> > it makes sense to do something like this and just defauilt bitOrder for
> > VariableRuntimeData's to MSBF, as from the examples of lookAhead that
> I've
> > seen involve looking into a hexBinary element.
> >
> > Thoughts?
> >
> > Josh
> >
>
Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Posted by "Adams, Joshua" <ja...@owlcyberdefense.com>.
Out of curiosity, would look ahead work at all if it was used as the default value of a regular defineVariable statement or is that not allowed? The spec says that the defaultValue of a defineVariable statement cannot reference the infoset, but using a lookAhead is a little different than referencing a forward path.
Josh
________________________________
From: Mike Beckerle <mb...@apache.org>
Sent: Tuesday, November 23, 2021 3:27 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Presumably the same issue would arise if dfdlx:lookahead was called from a
setVariable annotation on a sequence also.
I happen to know there is a requirement for this to reach into data at
boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
matter for this function.
I claim the right thing is for the bitOrder to come from the model group on
which the newVariableInstance or setVariable is residing.
My reasoning is that if you call dfdlx:lookAhead from the choiceDispatchKey
expression of a choice, the bitOrder is that of the choice model group.
That's how it works today.
There should be no difference between expressing a dfdl:choiceDispatchKey
that directly uses a dfdlx:lookAhead call, and one that does so by way of a
new variable instance.
<xs:choice>
<xs:annotation>
<xs:appinfo ...>
<dfdl:newVariableInstance ref="a:flag" defaultValue='{
dfdlx:lookAhead(....) }'/>
<dfdl:choice choiceDispatchKey='{ $a:flag }'/>
</xs:appinfo>
</xs:annotation>
.... choice branches here
</xs:choice>
So I think the right answer is to just require the enclosing model group's
model-group runtime data object be passed down to the newVariableInstance
or setVariable primitive parser/unparser.
On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua <ja...@owlcyberdefense.com>
wrote:
> I've been taking a look at this bug and am trying to figure out what the
> correct behavior for this situation would be. Currently, attempting to
> call dfdlx:lookAhead as part of an expression for the default value of a
> newVariableInstance will result in a UsageError being thrown when
> attempting to get the bitOrder, as the VariableRuntimeData is descended
> from NonTermRuntimeData.
>
> It doesn't seem unreasonable to want to use lookAhead from inside a
> newVariableInstance, but I'm not sure there is an easy way to get the
> parent sequence/element that contains the annotation where the
> newVariableInstance is defined. I noticed that for elements when the
> expression for byteOrder isn't defined, it just defaults to BigEndian with
> comments stating that this likely means that the element is likely
> hexBinary, in which case the byte order should be ignored. I don't know if
> it makes sense to do something like this and just defauilt bitOrder for
> VariableRuntimeData's to MSBF, as from the examples of lookAhead that I've
> seen involve looking into a hexBinary element.
>
> Thoughts?
>
> Josh
>
Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Posted by Mike Beckerle <mb...@apache.org>.
I took a look at this.
I believe the NewVariableInstanceStart and NewVariableInstanceEnd are
constructed inside a call to def gram(term: Term), and endGram(term: Term)
respectively.
You can pass this TermRuntimeData down to the
NewVariableInstanceStartParser and NewVariableInstanceStartUnparser.
Then, if you look at that parser and unparser, they currently have only one
arg named "context" which overrides the base context method, and the
variableRuntimeData is passed as this context arg.
If you rename that existing arg to vrd, pass the termRuntimeData as an
additional arg, and make that added termRuntimeData arg be the context
override, then I think that's the only thing you have to do. When the
expression is evaluated, it uses the context as part of the state, to
retrieve things like bitOrder, etc.
On Wed, Nov 24, 2021 at 2:03 PM Mike Beckerle <mb...@apache.org> wrote:
> bitOrder can't have an expression. Only byte order.
>
> On Wed, Nov 24, 2021 at 1:26 PM Adams, Joshua <ja...@owlcyberdefense.com>
> wrote:
>
>> I've been looking at this some more today and I'm not seeing an easy way
>> to get the enclosing sequence/group's bitOrder and apply it to the
>> NewVariableInstance runtime data. NVI has access to the
>> AnnotatedSchemaComponent, which does have access to enclosing elements, but
>> I don't think there is any access to the runtime data for the enclosing
>> element. I suppose it would be possible to read the raw attributes from
>> enclosing element but I'm not sure that would work well if bitOrder is
>> defined as an expression.
>>
>> Josh
>> ------------------------------
>> *From:* Mike Beckerle <mb...@apache.org>
>> *Sent:* Tuesday, November 23, 2021 3:27 PM
>> *To:* dev@daffodil.apache.org <de...@daffodil.apache.org>
>> *Subject:* Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
>>
>> Presumably the same issue would arise if dfdlx:lookahead was called from a
>> setVariable annotation on a sequence also.
>>
>> I happen to know there is a requirement for this to reach into data at
>> boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
>> matter for this function.
>>
>> I claim the right thing is for the bitOrder to come from the model group
>> on
>> which the newVariableInstance or setVariable is residing.
>>
>> My reasoning is that if you call dfdlx:lookAhead from the
>> choiceDispatchKey
>> expression of a choice, the bitOrder is that of the choice model group.
>> That's how it works today.
>> There should be no difference between expressing a dfdl:choiceDispatchKey
>> that directly uses a dfdlx:lookAhead call, and one that does so by way of
>> a
>> new variable instance.
>>
>> <xs:choice>
>> <xs:annotation>
>> <xs:appinfo ...>
>> <dfdl:newVariableInstance ref="a:flag" defaultValue='{
>> dfdlx:lookAhead(....) }'/>
>> <dfdl:choice choiceDispatchKey='{ $a:flag }'/>
>> </xs:appinfo>
>> </xs:annotation>
>> .... choice branches here
>> </xs:choice>
>>
>> So I think the right answer is to just require the enclosing model group's
>> model-group runtime data object be passed down to the newVariableInstance
>> or setVariable primitive parser/unparser.
>>
>> On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua <jadams@owlcyberdefense.com
>> >
>> wrote:
>>
>> > I've been taking a look at this bug and am trying to figure out what the
>> > correct behavior for this situation would be. Currently, attempting to
>> > call dfdlx:lookAhead as part of an expression for the default value of a
>> > newVariableInstance will result in a UsageError being thrown when
>> > attempting to get the bitOrder, as the VariableRuntimeData is descended
>> > from NonTermRuntimeData.
>> >
>> > It doesn't seem unreasonable to want to use lookAhead from inside a
>> > newVariableInstance, but I'm not sure there is an easy way to get the
>> > parent sequence/element that contains the annotation where the
>> > newVariableInstance is defined. I noticed that for elements when the
>> > expression for byteOrder isn't defined, it just defaults to BigEndian
>> with
>> > comments stating that this likely means that the element is likely
>> > hexBinary, in which case the byte order should be ignored. I don't
>> know if
>> > it makes sense to do something like this and just defauilt bitOrder for
>> > VariableRuntimeData's to MSBF, as from the examples of lookAhead that
>> I've
>> > seen involve looking into a hexBinary element.
>> >
>> > Thoughts?
>> >
>> > Josh
>> >
>>
>
Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Posted by Mike Beckerle <mb...@apache.org>.
bitOrder can't have an expression. Only byte order.
On Wed, Nov 24, 2021 at 1:26 PM Adams, Joshua <ja...@owlcyberdefense.com>
wrote:
> I've been looking at this some more today and I'm not seeing an easy way
> to get the enclosing sequence/group's bitOrder and apply it to the
> NewVariableInstance runtime data. NVI has access to the
> AnnotatedSchemaComponent, which does have access to enclosing elements, but
> I don't think there is any access to the runtime data for the enclosing
> element. I suppose it would be possible to read the raw attributes from
> enclosing element but I'm not sure that would work well if bitOrder is
> defined as an expression.
>
> Josh
> ------------------------------
> *From:* Mike Beckerle <mb...@apache.org>
> *Sent:* Tuesday, November 23, 2021 3:27 PM
> *To:* dev@daffodil.apache.org <de...@daffodil.apache.org>
> *Subject:* Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
>
> Presumably the same issue would arise if dfdlx:lookahead was called from a
> setVariable annotation on a sequence also.
>
> I happen to know there is a requirement for this to reach into data at
> boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
> matter for this function.
>
> I claim the right thing is for the bitOrder to come from the model group on
> which the newVariableInstance or setVariable is residing.
>
> My reasoning is that if you call dfdlx:lookAhead from the choiceDispatchKey
> expression of a choice, the bitOrder is that of the choice model group.
> That's how it works today.
> There should be no difference between expressing a dfdl:choiceDispatchKey
> that directly uses a dfdlx:lookAhead call, and one that does so by way of a
> new variable instance.
>
> <xs:choice>
> <xs:annotation>
> <xs:appinfo ...>
> <dfdl:newVariableInstance ref="a:flag" defaultValue='{
> dfdlx:lookAhead(....) }'/>
> <dfdl:choice choiceDispatchKey='{ $a:flag }'/>
> </xs:appinfo>
> </xs:annotation>
> .... choice branches here
> </xs:choice>
>
> So I think the right answer is to just require the enclosing model group's
> model-group runtime data object be passed down to the newVariableInstance
> or setVariable primitive parser/unparser.
>
> On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua <ja...@owlcyberdefense.com>
> wrote:
>
> > I've been taking a look at this bug and am trying to figure out what the
> > correct behavior for this situation would be. Currently, attempting to
> > call dfdlx:lookAhead as part of an expression for the default value of a
> > newVariableInstance will result in a UsageError being thrown when
> > attempting to get the bitOrder, as the VariableRuntimeData is descended
> > from NonTermRuntimeData.
> >
> > It doesn't seem unreasonable to want to use lookAhead from inside a
> > newVariableInstance, but I'm not sure there is an easy way to get the
> > parent sequence/element that contains the annotation where the
> > newVariableInstance is defined. I noticed that for elements when the
> > expression for byteOrder isn't defined, it just defaults to BigEndian
> with
> > comments stating that this likely means that the element is likely
> > hexBinary, in which case the byte order should be ignored. I don't know
> if
> > it makes sense to do something like this and just defauilt bitOrder for
> > VariableRuntimeData's to MSBF, as from the examples of lookAhead that
> I've
> > seen involve looking into a hexBinary element.
> >
> > Thoughts?
> >
> > Josh
> >
>
Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Posted by "Adams, Joshua" <ja...@owlcyberdefense.com>.
I've been looking at this some more today and I'm not seeing an easy way to get the enclosing sequence/group's bitOrder and apply it to the NewVariableInstance runtime data. NVI has access to the AnnotatedSchemaComponent, which does have access to enclosing elements, but I don't think there is any access to the runtime data for the enclosing element. I suppose it would be possible to read the raw attributes from enclosing element but I'm not sure that would work well if bitOrder is defined as an expression.
Josh
________________________________
From: Mike Beckerle <mb...@apache.org>
Sent: Tuesday, November 23, 2021 3:27 PM
To: dev@daffodil.apache.org <de...@daffodil.apache.org>
Subject: Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Presumably the same issue would arise if dfdlx:lookahead was called from a
setVariable annotation on a sequence also.
I happen to know there is a requirement for this to reach into data at
boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
matter for this function.
I claim the right thing is for the bitOrder to come from the model group on
which the newVariableInstance or setVariable is residing.
My reasoning is that if you call dfdlx:lookAhead from the choiceDispatchKey
expression of a choice, the bitOrder is that of the choice model group.
That's how it works today.
There should be no difference between expressing a dfdl:choiceDispatchKey
that directly uses a dfdlx:lookAhead call, and one that does so by way of a
new variable instance.
<xs:choice>
<xs:annotation>
<xs:appinfo ...>
<dfdl:newVariableInstance ref="a:flag" defaultValue='{
dfdlx:lookAhead(....) }'/>
<dfdl:choice choiceDispatchKey='{ $a:flag }'/>
</xs:appinfo>
</xs:annotation>
.... choice branches here
</xs:choice>
So I think the right answer is to just require the enclosing model group's
model-group runtime data object be passed down to the newVariableInstance
or setVariable primitive parser/unparser.
On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua <ja...@owlcyberdefense.com>
wrote:
> I've been taking a look at this bug and am trying to figure out what the
> correct behavior for this situation would be. Currently, attempting to
> call dfdlx:lookAhead as part of an expression for the default value of a
> newVariableInstance will result in a UsageError being thrown when
> attempting to get the bitOrder, as the VariableRuntimeData is descended
> from NonTermRuntimeData.
>
> It doesn't seem unreasonable to want to use lookAhead from inside a
> newVariableInstance, but I'm not sure there is an easy way to get the
> parent sequence/element that contains the annotation where the
> newVariableInstance is defined. I noticed that for elements when the
> expression for byteOrder isn't defined, it just defaults to BigEndian with
> comments stating that this likely means that the element is likely
> hexBinary, in which case the byte order should be ignored. I don't know if
> it makes sense to do something like this and just defauilt bitOrder for
> VariableRuntimeData's to MSBF, as from the examples of lookAhead that I've
> seen involve looking into a hexBinary element.
>
> Thoughts?
>
> Josh
>
Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance
Posted by Mike Beckerle <mb...@apache.org>.
Presumably the same issue would arise if dfdlx:lookahead was called from a
setVariable annotation on a sequence also.
I happen to know there is a requirement for this to reach into data at
boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
matter for this function.
I claim the right thing is for the bitOrder to come from the model group on
which the newVariableInstance or setVariable is residing.
My reasoning is that if you call dfdlx:lookAhead from the choiceDispatchKey
expression of a choice, the bitOrder is that of the choice model group.
That's how it works today.
There should be no difference between expressing a dfdl:choiceDispatchKey
that directly uses a dfdlx:lookAhead call, and one that does so by way of a
new variable instance.
<xs:choice>
<xs:annotation>
<xs:appinfo ...>
<dfdl:newVariableInstance ref="a:flag" defaultValue='{
dfdlx:lookAhead(....) }'/>
<dfdl:choice choiceDispatchKey='{ $a:flag }'/>
</xs:appinfo>
</xs:annotation>
.... choice branches here
</xs:choice>
So I think the right answer is to just require the enclosing model group's
model-group runtime data object be passed down to the newVariableInstance
or setVariable primitive parser/unparser.
On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua <ja...@owlcyberdefense.com>
wrote:
> I've been taking a look at this bug and am trying to figure out what the
> correct behavior for this situation would be. Currently, attempting to
> call dfdlx:lookAhead as part of an expression for the default value of a
> newVariableInstance will result in a UsageError being thrown when
> attempting to get the bitOrder, as the VariableRuntimeData is descended
> from NonTermRuntimeData.
>
> It doesn't seem unreasonable to want to use lookAhead from inside a
> newVariableInstance, but I'm not sure there is an easy way to get the
> parent sequence/element that contains the annotation where the
> newVariableInstance is defined. I noticed that for elements when the
> expression for byteOrder isn't defined, it just defaults to BigEndian with
> comments stating that this likely means that the element is likely
> hexBinary, in which case the byte order should be ignored. I don't know if
> it makes sense to do something like this and just defauilt bitOrder for
> VariableRuntimeData's to MSBF, as from the examples of lookAhead that I've
> seen involve looking into a hexBinary element.
>
> Thoughts?
>
> Josh
>