You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@daffodil.apache.org by Christofer Dutz <ch...@c-ware.de> on 2019/05/08 14:36:34 UTC
Some sort of "abstract" types?
HI,
I am currently working on a first code-generator for my DFDL schemas … as implementing the full spec is completely out of question as it’s just far to feature-rich and hereby complex to implement, I am currently refactoring my schemas to allow everything I need, but as simple as possible. While I was able to come up with solutions for most things, one thing is a little annoying:
We have an element like this:
<xs:element name="data" type="xs:hexBinary"
dfdl:byteOrder="bigEndian" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit"
dfdl:length="{../../parameterLength}"/>
We moved the definition of all types to the top in “simpleType” elements and simply reference “plc4x:uint8” for example, however we can’t do that with these as we can’t override the “length” later on …
I would have preferred something like this:
<xs:simpleType name="data" dfdl:lengthUnits="bytes" dfdl:length="1" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
And then override the length where needed:
<xs:element name="data" type="plc4x:data" dfdl:length="{../../parameterLength}"/>
Are there any options to do such a thing and I just haven’t found them yet?
Chris
Re: Some sort of "abstract" types?
Posted by "Beckerle, Mike" <mb...@tresys.com>.
Not at all a waste of my time. No worries. I was able to get you to continue trying other alternatives, and we got you to a solution, so all goodness.
I don't think anything changed here since 2.0.0, but that's so far back now....
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 3:46:01 PM
To: users@daffodil.apache.org
Subject: Re: Some sort of "abstract" types?
Hi Mike,
please excuse the confusion … I could have sworn that I had tried exactly that in the past and I got errors for not defining the length in the dataN type and getting an error when overriding it when using it.
I just gave it a spin the way you suggested and it worked.
Was something changed since 2.0.0 or so in this sector (Think it was around that time that I tried it)?
But from the way it looks, it now works exactly as I would have hoped and I could have sworn I tried that path before.
If it was my mistake, I would like to apologize for having wasted your time.
Thanks a lot … now I can define my schemas the way I need them to generate them … well at least till I run into the next thing … code generation isn’t that trivial.
Hopefully our work in the PLC4X project will help here some time.
Chris
Von: "Beckerle, Mike" <mb...@tresys.com>
Antworten an: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Datum: Mittwoch, 8. Mai 2019 um 18:16
An: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Betreff: Re: Some sort of "abstract" types?
Please help me understand what you need better.
As co-chair on the DFDL workgroup, if there is something you need to be able to express, and there truly is no good way to do it, then I would consider that a DFDL flaw that we should really examine. If we (by we, I mean the DFDL workgroup, and the DFDL user community) decide it's a flaw, we would propose and implement a fix in Daffodil, then propose it for inclusion in the DFDL standard. So even if DFDL can't do something today, doesn't mean it wouldn't get fixed fairly quickly in Daffodil.
You are looking for abstract types. I'm not sure what you are trying to be abstract about.
One can write an element, giving only where the length comes from.
<xs:element name="data" type="tns:dataN"
dfdl:length="{../../parameterLength}"/>
To me, dataN here is an abstract type where you must provide length property to make it concrete. All the other properties are fixed in the type definition, or in the lexical environment (top level format) surrounding that type definition.
You can even make it reusable with the length coming from a known relative location.
<xs:simpleType name="dataParameterizedLength"
dfdl:length="{../../parameterLength}">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
<xs:element name="data" type="tns:dataParameterizedLength"/>
In this case the type "dataParameterizedLength" is a concrete type, where dataN is the abstract base.
You or you can use variables to guide whether the length is parameterized or fixed:
<xs:simpleType name="dataParameterizedLength"
dfdl:length="{ if ($tns:useFixedLengths) then 1
else ../../parameterLength }">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
What you can't do is define a type completely with all representation types, and then evolve them and change your mind in derived types. You have to anticipate the need to override, and create an abstract type that lacks the properties you want to specify separately.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 11:39 AM
To: users@daffodil.apache.org
Subject: Re: Some sort of "abstract" types?
Hi Mike,
but if I want to read a number of bytes which size is defined by another property, then things get complicated/impossible.
I am currently trying to separate the reading of bits/bytes which is sort of language dependent into a language-dependent IO module
and have the structure of the message use these globally defined types to generate the datatypes and the parsers/serializers.
So for now I’ll probably have to give up the ability to run my schemas in daffodil …
I would have liked to continue as I really liked things like the test-suite stuff and such, but right now it’s this which is causing trouble …
perhaps we’ll come up with exceptions for hexByte and string and will allow only there to be defined inline and force the rest to be imported.
Chris
Von: "Beckerle, Mike" <mb...@tresys.com>
Antworten an: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Datum: Mittwoch, 8. Mai 2019 um 17:18
An: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Betreff: Re: Some sort of "abstract" types?
Alas, the DFDL Workgroup made a "conservative design choice" a long while back to disallow overlapping properties between referencer and referencee, where those referencing scenarios are:
element -> type def
elementRef -> element decl
groupRef -> model group of group def
simple type def -> simple base type def
The rationale for this was that we could always loosen this restriction in the future, but some of the committee members believed this restriction would simplify implementation complexity.
To achieve roughly what you want you have to do:
<xs:simpleType name="dataN" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
<xs:simpleType name="data1" dfdl:length="1">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
Then at your points of use of the type, you use data1 or dataN accordingly.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 10:36:34 AM
To: users@daffodil.apache.org
Subject: Some sort of "abstract" types?
HI,
I am currently working on a first code-generator for my DFDL schemas … as implementing the full spec is completely out of question as it’s just far to feature-rich and hereby complex to implement, I am currently refactoring my schemas to allow everything I need, but as simple as possible. While I was able to come up with solutions for most things, one thing is a little annoying:
We have an element like this:
<xs:element name="data" type="xs:hexBinary"
dfdl:byteOrder="bigEndian" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit"
dfdl:length="{../../parameterLength}"/>
We moved the definition of all types to the top in “simpleType” elements and simply reference “plc4x:uint8” for example, however we can’t do that with these as we can’t override the “length” later on …
I would have preferred something like this:
<xs:simpleType name="data" dfdl:lengthUnits="bytes" dfdl:length="1" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
And then override the length where needed:
<xs:element name="data" type="plc4x:data" dfdl:length="{../../parameterLength}"/>
Are there any options to do such a thing and I just haven’t found them yet?
Chris
Re: Some sort of "abstract" types?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Mike,
please excuse the confusion … I could have sworn that I had tried exactly that in the past and I got errors for not defining the length in the dataN type and getting an error when overriding it when using it.
I just gave it a spin the way you suggested and it worked.
Was something changed since 2.0.0 or so in this sector (Think it was around that time that I tried it)?
But from the way it looks, it now works exactly as I would have hoped and I could have sworn I tried that path before.
If it was my mistake, I would like to apologize for having wasted your time.
Thanks a lot … now I can define my schemas the way I need them to generate them … well at least till I run into the next thing … code generation isn’t that trivial.
Hopefully our work in the PLC4X project will help here some time.
Chris
Von: "Beckerle, Mike" <mb...@tresys.com>
Antworten an: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Datum: Mittwoch, 8. Mai 2019 um 18:16
An: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Betreff: Re: Some sort of "abstract" types?
Please help me understand what you need better.
As co-chair on the DFDL workgroup, if there is something you need to be able to express, and there truly is no good way to do it, then I would consider that a DFDL flaw that we should really examine. If we (by we, I mean the DFDL workgroup, and the DFDL user community) decide it's a flaw, we would propose and implement a fix in Daffodil, then propose it for inclusion in the DFDL standard. So even if DFDL can't do something today, doesn't mean it wouldn't get fixed fairly quickly in Daffodil.
You are looking for abstract types. I'm not sure what you are trying to be abstract about.
One can write an element, giving only where the length comes from.
<xs:element name="data" type="tns:dataN"
dfdl:length="{../../parameterLength}"/>
To me, dataN here is an abstract type where you must provide length property to make it concrete. All the other properties are fixed in the type definition, or in the lexical environment (top level format) surrounding that type definition.
You can even make it reusable with the length coming from a known relative location.
<xs:simpleType name="dataParameterizedLength"
dfdl:length="{../../parameterLength}">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
<xs:element name="data" type="tns:dataParameterizedLength"/>
In this case the type "dataParameterizedLength" is a concrete type, where dataN is the abstract base.
You or you can use variables to guide whether the length is parameterized or fixed:
<xs:simpleType name="dataParameterizedLength"
dfdl:length="{ if ($tns:useFixedLengths) then 1
else ../../parameterLength }">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
What you can't do is define a type completely with all representation types, and then evolve them and change your mind in derived types. You have to anticipate the need to override, and create an abstract type that lacks the properties you want to specify separately.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 11:39 AM
To: users@daffodil.apache.org
Subject: Re: Some sort of "abstract" types?
Hi Mike,
but if I want to read a number of bytes which size is defined by another property, then things get complicated/impossible.
I am currently trying to separate the reading of bits/bytes which is sort of language dependent into a language-dependent IO module
and have the structure of the message use these globally defined types to generate the datatypes and the parsers/serializers.
So for now I’ll probably have to give up the ability to run my schemas in daffodil …
I would have liked to continue as I really liked things like the test-suite stuff and such, but right now it’s this which is causing trouble …
perhaps we’ll come up with exceptions for hexByte and string and will allow only there to be defined inline and force the rest to be imported.
Chris
Von: "Beckerle, Mike" <mb...@tresys.com>
Antworten an: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Datum: Mittwoch, 8. Mai 2019 um 17:18
An: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Betreff: Re: Some sort of "abstract" types?
Alas, the DFDL Workgroup made a "conservative design choice" a long while back to disallow overlapping properties between referencer and referencee, where those referencing scenarios are:
element -> type def
elementRef -> element decl
groupRef -> model group of group def
simple type def -> simple base type def
The rationale for this was that we could always loosen this restriction in the future, but some of the committee members believed this restriction would simplify implementation complexity.
To achieve roughly what you want you have to do:
<xs:simpleType name="dataN" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
<xs:simpleType name="data1" dfdl:length="1">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
Then at your points of use of the type, you use data1 or dataN accordingly.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 10:36:34 AM
To: users@daffodil.apache.org
Subject: Some sort of "abstract" types?
HI,
I am currently working on a first code-generator for my DFDL schemas … as implementing the full spec is completely out of question as it’s just far to feature-rich and hereby complex to implement, I am currently refactoring my schemas to allow everything I need, but as simple as possible. While I was able to come up with solutions for most things, one thing is a little annoying:
We have an element like this:
<xs:element name="data" type="xs:hexBinary"
dfdl:byteOrder="bigEndian" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit"
dfdl:length="{../../parameterLength}"/>
We moved the definition of all types to the top in “simpleType” elements and simply reference “plc4x:uint8” for example, however we can’t do that with these as we can’t override the “length” later on …
I would have preferred something like this:
<xs:simpleType name="data" dfdl:lengthUnits="bytes" dfdl:length="1" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
And then override the length where needed:
<xs:element name="data" type="plc4x:data" dfdl:length="{../../parameterLength}"/>
Are there any options to do such a thing and I just haven’t found them yet?
Chris
Re: Some sort of "abstract" types?
Posted by "Beckerle, Mike" <mb...@tresys.com>.
Please help me understand what you need better.
As co-chair on the DFDL workgroup, if there is something you need to be able to express, and there truly is no good way to do it, then I would consider that a DFDL flaw that we should really examine. If we (by we, I mean the DFDL workgroup, and the DFDL user community) decide it's a flaw, we would propose and implement a fix in Daffodil, then propose it for inclusion in the DFDL standard. So even if DFDL can't do something today, doesn't mean it wouldn't get fixed fairly quickly in Daffodil.
You are looking for abstract types. I'm not sure what you are trying to be abstract about.
One can write an element, giving only where the length comes from.
<xs:element name="data" type="tns:dataN"
dfdl:length="{../../parameterLength}"/>
To me, dataN here is an abstract type where you must provide length property to make it concrete. All the other properties are fixed in the type definition, or in the lexical environment (top level format) surrounding that type definition.
You can even make it reusable with the length coming from a known relative location.
<xs:simpleType name="dataParameterizedLength"
dfdl:length="{../../parameterLength}">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
<xs:element name="data" type="tns:dataParameterizedLength"/>
In this case the type "dataParameterizedLength" is a concrete type, where dataN is the abstract base.
You or you can use variables to guide whether the length is parameterized or fixed:
<xs:simpleType name="dataParameterizedLength"
dfdl:length="{ if ($tns:useFixedLengths) then 1
else ../../parameterLength }">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
What you can't do is define a type completely with all representation types, and then evolve them and change your mind in derived types. You have to anticipate the need to override, and create an abstract type that lacks the properties you want to specify separately.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 11:39 AM
To: users@daffodil.apache.org
Subject: Re: Some sort of "abstract" types?
Hi Mike,
but if I want to read a number of bytes which size is defined by another property, then things get complicated/impossible.
I am currently trying to separate the reading of bits/bytes which is sort of language dependent into a language-dependent IO module
and have the structure of the message use these globally defined types to generate the datatypes and the parsers/serializers.
So for now I’ll probably have to give up the ability to run my schemas in daffodil …
I would have liked to continue as I really liked things like the test-suite stuff and such, but right now it’s this which is causing trouble …
perhaps we’ll come up with exceptions for hexByte and string and will allow only there to be defined inline and force the rest to be imported.
Chris
Von: "Beckerle, Mike" <mb...@tresys.com>
Antworten an: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Datum: Mittwoch, 8. Mai 2019 um 17:18
An: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Betreff: Re: Some sort of "abstract" types?
Alas, the DFDL Workgroup made a "conservative design choice" a long while back to disallow overlapping properties between referencer and referencee, where those referencing scenarios are:
element -> type def
elementRef -> element decl
groupRef -> model group of group def
simple type def -> simple base type def
The rationale for this was that we could always loosen this restriction in the future, but some of the committee members believed this restriction would simplify implementation complexity.
To achieve roughly what you want you have to do:
<xs:simpleType name="dataN" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
<xs:simpleType name="data1" dfdl:length="1">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
Then at your points of use of the type, you use data1 or dataN accordingly.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 10:36:34 AM
To: users@daffodil.apache.org
Subject: Some sort of "abstract" types?
HI,
I am currently working on a first code-generator for my DFDL schemas … as implementing the full spec is completely out of question as it’s just far to feature-rich and hereby complex to implement, I am currently refactoring my schemas to allow everything I need, but as simple as possible. While I was able to come up with solutions for most things, one thing is a little annoying:
We have an element like this:
<xs:element name="data" type="xs:hexBinary"
dfdl:byteOrder="bigEndian" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit"
dfdl:length="{../../parameterLength}"/>
We moved the definition of all types to the top in “simpleType” elements and simply reference “plc4x:uint8” for example, however we can’t do that with these as we can’t override the “length” later on …
I would have preferred something like this:
<xs:simpleType name="data" dfdl:lengthUnits="bytes" dfdl:length="1" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
And then override the length where needed:
<xs:element name="data" type="plc4x:data" dfdl:length="{../../parameterLength}"/>
Are there any options to do such a thing and I just haven’t found them yet?
Chris
Re: Some sort of "abstract" types?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Mike,
but if I want to read a number of bytes which size is defined by another property, then things get complicated/impossible.
I am currently trying to separate the reading of bits/bytes which is sort of language dependent into a language-dependent IO module
and have the structure of the message use these globally defined types to generate the datatypes and the parsers/serializers.
So for now I’ll probably have to give up the ability to run my schemas in daffodil …
I would have liked to continue as I really liked things like the test-suite stuff and such, but right now it’s this which is causing trouble …
perhaps we’ll come up with exceptions for hexByte and string and will allow only there to be defined inline and force the rest to be imported.
Chris
Von: "Beckerle, Mike" <mb...@tresys.com>
Antworten an: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Datum: Mittwoch, 8. Mai 2019 um 17:18
An: "users@daffodil.apache.org" <us...@daffodil.apache.org>
Betreff: Re: Some sort of "abstract" types?
Alas, the DFDL Workgroup made a "conservative design choice" a long while back to disallow overlapping properties between referencer and referencee, where those referencing scenarios are:
element -> type def
elementRef -> element decl
groupRef -> model group of group def
simple type def -> simple base type def
The rationale for this was that we could always loosen this restriction in the future, but some of the committee members believed this restriction would simplify implementation complexity.
To achieve roughly what you want you have to do:
<xs:simpleType name="dataN" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
<xs:simpleType name="data1" dfdl:length="1">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
Then at your points of use of the type, you use data1 or dataN accordingly.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 10:36:34 AM
To: users@daffodil.apache.org
Subject: Some sort of "abstract" types?
HI,
I am currently working on a first code-generator for my DFDL schemas … as implementing the full spec is completely out of question as it’s just far to feature-rich and hereby complex to implement, I am currently refactoring my schemas to allow everything I need, but as simple as possible. While I was able to come up with solutions for most things, one thing is a little annoying:
We have an element like this:
<xs:element name="data" type="xs:hexBinary"
dfdl:byteOrder="bigEndian" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit"
dfdl:length="{../../parameterLength}"/>
We moved the definition of all types to the top in “simpleType” elements and simply reference “plc4x:uint8” for example, however we can’t do that with these as we can’t override the “length” later on …
I would have preferred something like this:
<xs:simpleType name="data" dfdl:lengthUnits="bytes" dfdl:length="1" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
And then override the length where needed:
<xs:element name="data" type="plc4x:data" dfdl:length="{../../parameterLength}"/>
Are there any options to do such a thing and I just haven’t found them yet?
Chris
Re: Some sort of "abstract" types?
Posted by "Beckerle, Mike" <mb...@tresys.com>.
Alas, the DFDL Workgroup made a "conservative design choice" a long while back to disallow overlapping properties between referencer and referencee, where those referencing scenarios are:
element -> type def
elementRef -> element decl
groupRef -> model group of group def
simple type def -> simple base type def
The rationale for this was that we could always loosen this restriction in the future, but some of the committee members believed this restriction would simplify implementation complexity.
To achieve roughly what you want you have to do:
<xs:simpleType name="dataN" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
<xs:simpleType name="data1" dfdl:length="1">
<xs:restriction base="tns:dataN"/>
</xs:simpleType>
Then at your points of use of the type, you use data1 or dataN accordingly.
________________________________
From: Christofer Dutz <ch...@c-ware.de>
Sent: Wednesday, May 8, 2019 10:36:34 AM
To: users@daffodil.apache.org
Subject: Some sort of "abstract" types?
HI,
I am currently working on a first code-generator for my DFDL schemas … as implementing the full spec is completely out of question as it’s just far to feature-rich and hereby complex to implement, I am currently refactoring my schemas to allow everything I need, but as simple as possible. While I was able to come up with solutions for most things, one thing is a little annoying:
We have an element like this:
<xs:element name="data" type="xs:hexBinary"
dfdl:byteOrder="bigEndian" dfdl:lengthUnits="bytes" dfdl:lengthKind="explicit"
dfdl:length="{../../parameterLength}"/>
We moved the definition of all types to the top in “simpleType” elements and simply reference “plc4x:uint8” for example, however we can’t do that with these as we can’t override the “length” later on …
I would have preferred something like this:
<xs:simpleType name="data" dfdl:lengthUnits="bytes" dfdl:length="1" dfdl:lengthKind="explicit">
<xs:restriction base="xs:hexBinary"/>
</xs:simpleType>
And then override the length where needed:
<xs:element name="data" type="plc4x:data" dfdl:length="{../../parameterLength}"/>
Are there any options to do such a thing and I just haven’t found them yet?
Chris