You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by Marshall Miller <ma...@quarksecurity.com> on 2018/01/11 19:34:46 UTC
Help implementing schema for SISO-J
Hello,
I am trying to implement a schema to describe a LINK-16 message
embedded within a DIS Signal PDU as described by SISO-STD-002 [1],
paragraph 5.2.2.9 and table 5.2.3; commonly referred to as SISO-J.
The problem is that fields are packed into 32 bit words with big
endian byte order, and little endian bitfield order. It is currently
an error to specify a byteOrder of bigEndian with a bitOrder of
leastSignificantBitFirst. Also, I have not found a way to set word
size; it appears that word size is implicitly set to the size of each
field.
Following is a simple example to demonstrate how fields need to be
interpreted. It is just an example and does not represent any part of
a SISO-J message.
Data is comprised of 3 unsigned integer bitfields:
slot1 is 12 bits
slot2 is 32 bits
slot3 is 20 bits
Given an input with the following bytes in hex:
01 23 45 67 89 AB CD EF
Parsed results need to be (in hex):
slot1: 0567
slot2: DEF01234
slot3: 089ABC
Explanation of results:
Since the bitfields need to consume data in a leastSignificantBitFirst
order, it is easier to represent the input in right-to-left order
where bits are consumed starting at the right and moving toward the
left. Since words are 4 bytes, and in big endian order, the lowest
bit in the 4th byte is the first bit to be consumed. Reordering the
input data and consuming data from right to left yields the correct
field boundaries.
89 AB CD EF 01 23 45 67 parse fields right to left
05 06 07 08 01 02 03 04 input byte position
slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles,
and slot3 consumes the left 5 nibbles.
Could you please let me know if there is a way to parse fields in this manner?
Marshall Miller
[1] https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=43048&PortalId=0&TabId=105
Re: Help implementing schema for SISO-J
Posted by Mike Beckerle <mb...@tresys.com>.
Here's our roadmap: https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upcoming+Releases
Release 2.2 is what has this feature.
If you can contribute code to Daffodil that will also help us with our agenda to convert Daffodil into an Apache project. Right now we're an "Incubator" project at Apache, and can't graduate to a full project status until we have some community diversity where developers span more organizations than just Tresys.
________________________________
From: Marshall Miller <ma...@quarksecurity.com>
Sent: Friday, January 12, 2018 11:40:39 AM
To: dev@daffodil.apache.org
Subject: Re: Help implementing schema for SISO-J
The first portion of one of these Signal PDUs is in BE+MSBF order up
to and including a header for the LINK-16 portion. Then there is the
BE+LSBF data which is a JTIDS header followed by the contents of the J
message.
The pre/post processing feature does seem like a better hammer for the
problem. I believe we could contribute a pre/post processor that can
handle swapping. Do you have an idea on when this new feature will
start to be implemented?
On Thu, Jan 11, 2018 at 8:11 PM, Mike Beckerle <mb...@tresys.com> wrote:
> The byte-order swapping sounds right Marshall.
>
>
> One of the features for DFDL/Daffodil for which there is much demand is a feature that enables pre-processing passes like what you are describing. Things like base64 encoding, line-folding, byte-order/word swapping, decompress, are all things people need to be able to do on the underlying data stream; not necessarily of the entire data stream, but on just the section of the data stream holding the representation of a specific element within a greater format. E.g., VMF messages are often carried as a compressed payload within a MIL-STD-2045 envelope. Internet messages have attachments that are base64 encoded commonly, etc etc.
>
>
> In your case, this word/byte swapping would be applicable to the SISO-J headers, but would it apply to the Link16 data being carried by it? If it applies only to the SISO-J packaging part, then you'd need this feature.
>
>
> The current proposal for this feature (which I'm calling the "layering" or "base64" feature) was actually circulated on this mailing list last week for initial feedback. Implementing this feature in Daffodil is within our 6 month roadmap/plan.
>
>
> I like this better than trying to come up with an endless proliferation of word, byte, and bit order properties, the interactions of which are not very orthogonal.
>
>
> As for the way this or any other feature gets incorporated into the DFDL language spec. Basically the cycle is 1) propose and discuss feature 2) prototype feature in Daffodil 3) get experience with feature 4) officially propose for inclusion in DFDL. That's if you are working with Daffodil. If you worked for IBM you would perhaps prototype in their implementation.
>
>
> If we implement base64 and/or compress maybe we could get you to develop the word-swapping pre/post processor which would would work analogously in Daffodil?
>
> ________________________________
> From: Marshall Miller <ma...@quarksecurity.com>
> Sent: Thursday, January 11, 2018 5:18:26 PM
> To: dev@daffodil.apache.org
> Subject: Re: Help implementing schema for SISO-J
>
> Also, it occurred to me that it might be possible to perform a stream
> transformation on the data before it is parsed. If each 32 bit word
> were byte-swapped before parsing fields, then I believe the fields
> could be properly parsed in littleEndian byte order with LSBF
> bitOrder.
>
> On Thu, Jan 11, 2018 at 4:50 PM, Marshall Miller
> <ma...@quarksecurity.com> wrote:
>> Mike,
>>
>> Thanks for the helpful and quick response. I was worried we might
>> have to extract the fields with mathematical operations as you
>> described.
>>
>> Hopefully the DFDL Workgroup will add support for this type of packing
>> so that LINK-16 schemas can be reused in SISO-J messages. When does
>> the workgroup meet next? What do you think the likelihood is for a
>> change like this to be accepted?
>>
>>
>> Marshall Miller
>>
>> On Thu, Jan 11, 2018 at 3:48 PM, Mike Beckerle <mb...@tresys.com> wrote:
>>> Marshall,
>>>
>>>
>>> Looks like we'll have to give some thought to whether bigEndian and LSBF can go together. We've had no actual examples of this until now. I will bring that up with the OGF DFDL Workgroup.
>>>
>>>
>>> But to deal with this today, these kinds of mixed bit order situations generally require use of the dfdl:inputValueCalc and dfdl:outputValueCalc properties.
>>>
>>>
>>> A simple example is here:
>>>
>>>
>>> https://github.com/rchampag/dfdl/blob/master/arinc429-label/localizer-deviation-label.dfdl.xsd
>>>
>>>
>>> This was for individual bits, but in your case they would be wider integers.
>>>
>>>
>>> For your 3 fields in the example I suggest you parse as 4 fields
>>>
>>>
>>> h_slot2LSBs - 20 bits
>>>
>>> h_slot1 - 12 bits
>>>
>>> h_slot3 - 20 bits
>>>
>>> h_slot2MSBs - 12 bits
>>>
>>>
>>> You would do this all bigEndian and mostSignificantBitFirst.
>>>
>>>
>>> The above would go into a hidden group so they wouldn't show up in the logical data.
>>>
>>>
>>> Then you would have 3 logical ordinary, not hidden elements
>>>
>>> slot1 dfdl:inputValueCalc='{ ../h_slot1 }'
>>>
>>> slot2 dfdl:inputValueCalc='{ ( ../h_slot2MSBs * dfdl:int('x80000' ) + ../h_slot2LSBs }'
>>>
>>> slot3 dfdl:inputValueCalc='{ ../h_slot3 }'
>>>
>>>
>>> dfdl:int('x80000') constructs a integer from a hex string, and I think that's the right value. Or maybe it's 'x100000'.
>>>
>>>
>>> You can also use math.pow function as in
>>>
>>>
>>> xs:int(math:pow(2, 19))
>>>
>>>
>>> (you need to add this namespace to your xs:schema element xmlns:math='http://www.w3.org/2005/xpath-functions/math')
>>>
>>>
>>> The tricky part is that for unparsing you must invert this using dfdl:outputValueCalc. So the 4 hidden elements would be like this:
>>>
>>>
>>> h_slot2LSBs dfdl:outputValueCalc="{ ../slot2 mod dfdl:int('x80000') }"
>>>
>>> h_slot1 dfdl:outputValueCalc='{ ../slot1 }'
>>>
>>> h_slot3 dfdl:outputValueCalc='{ ../slot3 }'
>>>
>>> h_slot2MSBs dfdl:outputValueCalc="{ ../slot2 idiv dfdl:int('x80000') }"
>>>
>>>
>>> or something like that. My details here are probably off by 1 or something.
>>>
>>>
>>> DFDL's Xpath-like expression language has the operators +, -, *, div, idiv, mod, and the math.pow() function. Alas there's no shift nor a bit-wise & and | operator. Generally it is possible to figure out how to achieve what you want by way of mathematical operations.
>>>
>>>
>>> Also note: If you want Link16 specific support we have a channel for FOUO email support which is daffodil-fouo-support@tresys.com.
>>>
>>>
>>> ...mike beckerle
>>>
>>> Tresys
>>>
>>>
>>>
>>> ________________________________
>>> From: Marshall Miller <ma...@quarksecurity.com>
>>> Sent: Thursday, January 11, 2018 2:34:46 PM
>>> To: dev@daffodil.apache.org
>>> Subject: Help implementing schema for SISO-J
>>>
>>> Hello,
>>>
>>> I am trying to implement a schema to describe a LINK-16 message
>>> embedded within a DIS Signal PDU as described by SISO-STD-002 [1],
>>> paragraph 5.2.2.9 and table 5.2.3; commonly referred to as SISO-J.
>>> The problem is that fields are packed into 32 bit words with big
>>> endian byte order, and little endian bitfield order. It is currently
>>> an error to specify a byteOrder of bigEndian with a bitOrder of
>>> leastSignificantBitFirst. Also, I have not found a way to set word
>>> size; it appears that word size is implicitly set to the size of each
>>> field.
>>>
>>> Following is a simple example to demonstrate how fields need to be
>>> interpreted. It is just an example and does not represent any part of
>>> a SISO-J message.
>>>
>>> Data is comprised of 3 unsigned integer bitfields:
>>> slot1 is 12 bits
>>> slot2 is 32 bits
>>> slot3 is 20 bits
>>>
>>> Given an input with the following bytes in hex:
>>> 01 23 45 67 89 AB CD EF
>>>
>>> Parsed results need to be (in hex):
>>> slot1: 0567
>>> slot2: DEF01234
>>> slot3: 089ABC
>>>
>>> Explanation of results:
>>>
>>> Since the bitfields need to consume data in a leastSignificantBitFirst
>>> order, it is easier to represent the input in right-to-left order
>>> where bits are consumed starting at the right and moving toward the
>>> left. Since words are 4 bytes, and in big endian order, the lowest
>>> bit in the 4th byte is the first bit to be consumed. Reordering the
>>> input data and consuming data from right to left yields the correct
>>> field boundaries.
>>>
>>> 89 AB CD EF 01 23 45 67 parse fields right to left
>>> 05 06 07 08 01 02 03 04 input byte position
>>>
>>> slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles,
>>> and slot3 consumes the left 5 nibbles.
>>>
>>> Could you please let me know if there is a way to parse fields in this manner?
>>>
>>>
>>>
>>> Marshall Miller
>>>
>>>
>>> [1] https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=43048&PortalId=0&TabId=105
Re: Help implementing schema for SISO-J
Posted by Marshall Miller <ma...@quarksecurity.com>.
The first portion of one of these Signal PDUs is in BE+MSBF order up
to and including a header for the LINK-16 portion. Then there is the
BE+LSBF data which is a JTIDS header followed by the contents of the J
message.
The pre/post processing feature does seem like a better hammer for the
problem. I believe we could contribute a pre/post processor that can
handle swapping. Do you have an idea on when this new feature will
start to be implemented?
On Thu, Jan 11, 2018 at 8:11 PM, Mike Beckerle <mb...@tresys.com> wrote:
> The byte-order swapping sounds right Marshall.
>
>
> One of the features for DFDL/Daffodil for which there is much demand is a feature that enables pre-processing passes like what you are describing. Things like base64 encoding, line-folding, byte-order/word swapping, decompress, are all things people need to be able to do on the underlying data stream; not necessarily of the entire data stream, but on just the section of the data stream holding the representation of a specific element within a greater format. E.g., VMF messages are often carried as a compressed payload within a MIL-STD-2045 envelope. Internet messages have attachments that are base64 encoded commonly, etc etc.
>
>
> In your case, this word/byte swapping would be applicable to the SISO-J headers, but would it apply to the Link16 data being carried by it? If it applies only to the SISO-J packaging part, then you'd need this feature.
>
>
> The current proposal for this feature (which I'm calling the "layering" or "base64" feature) was actually circulated on this mailing list last week for initial feedback. Implementing this feature in Daffodil is within our 6 month roadmap/plan.
>
>
> I like this better than trying to come up with an endless proliferation of word, byte, and bit order properties, the interactions of which are not very orthogonal.
>
>
> As for the way this or any other feature gets incorporated into the DFDL language spec. Basically the cycle is 1) propose and discuss feature 2) prototype feature in Daffodil 3) get experience with feature 4) officially propose for inclusion in DFDL. That's if you are working with Daffodil. If you worked for IBM you would perhaps prototype in their implementation.
>
>
> If we implement base64 and/or compress maybe we could get you to develop the word-swapping pre/post processor which would would work analogously in Daffodil?
>
> ________________________________
> From: Marshall Miller <ma...@quarksecurity.com>
> Sent: Thursday, January 11, 2018 5:18:26 PM
> To: dev@daffodil.apache.org
> Subject: Re: Help implementing schema for SISO-J
>
> Also, it occurred to me that it might be possible to perform a stream
> transformation on the data before it is parsed. If each 32 bit word
> were byte-swapped before parsing fields, then I believe the fields
> could be properly parsed in littleEndian byte order with LSBF
> bitOrder.
>
> On Thu, Jan 11, 2018 at 4:50 PM, Marshall Miller
> <ma...@quarksecurity.com> wrote:
>> Mike,
>>
>> Thanks for the helpful and quick response. I was worried we might
>> have to extract the fields with mathematical operations as you
>> described.
>>
>> Hopefully the DFDL Workgroup will add support for this type of packing
>> so that LINK-16 schemas can be reused in SISO-J messages. When does
>> the workgroup meet next? What do you think the likelihood is for a
>> change like this to be accepted?
>>
>>
>> Marshall Miller
>>
>> On Thu, Jan 11, 2018 at 3:48 PM, Mike Beckerle <mb...@tresys.com> wrote:
>>> Marshall,
>>>
>>>
>>> Looks like we'll have to give some thought to whether bigEndian and LSBF can go together. We've had no actual examples of this until now. I will bring that up with the OGF DFDL Workgroup.
>>>
>>>
>>> But to deal with this today, these kinds of mixed bit order situations generally require use of the dfdl:inputValueCalc and dfdl:outputValueCalc properties.
>>>
>>>
>>> A simple example is here:
>>>
>>>
>>> https://github.com/rchampag/dfdl/blob/master/arinc429-label/localizer-deviation-label.dfdl.xsd
>>>
>>>
>>> This was for individual bits, but in your case they would be wider integers.
>>>
>>>
>>> For your 3 fields in the example I suggest you parse as 4 fields
>>>
>>>
>>> h_slot2LSBs - 20 bits
>>>
>>> h_slot1 - 12 bits
>>>
>>> h_slot3 - 20 bits
>>>
>>> h_slot2MSBs - 12 bits
>>>
>>>
>>> You would do this all bigEndian and mostSignificantBitFirst.
>>>
>>>
>>> The above would go into a hidden group so they wouldn't show up in the logical data.
>>>
>>>
>>> Then you would have 3 logical ordinary, not hidden elements
>>>
>>> slot1 dfdl:inputValueCalc='{ ../h_slot1 }'
>>>
>>> slot2 dfdl:inputValueCalc='{ ( ../h_slot2MSBs * dfdl:int('x80000' ) + ../h_slot2LSBs }'
>>>
>>> slot3 dfdl:inputValueCalc='{ ../h_slot3 }'
>>>
>>>
>>> dfdl:int('x80000') constructs a integer from a hex string, and I think that's the right value. Or maybe it's 'x100000'.
>>>
>>>
>>> You can also use math.pow function as in
>>>
>>>
>>> xs:int(math:pow(2, 19))
>>>
>>>
>>> (you need to add this namespace to your xs:schema element xmlns:math='http://www.w3.org/2005/xpath-functions/math')
>>>
>>>
>>> The tricky part is that for unparsing you must invert this using dfdl:outputValueCalc. So the 4 hidden elements would be like this:
>>>
>>>
>>> h_slot2LSBs dfdl:outputValueCalc="{ ../slot2 mod dfdl:int('x80000') }"
>>>
>>> h_slot1 dfdl:outputValueCalc='{ ../slot1 }'
>>>
>>> h_slot3 dfdl:outputValueCalc='{ ../slot3 }'
>>>
>>> h_slot2MSBs dfdl:outputValueCalc="{ ../slot2 idiv dfdl:int('x80000') }"
>>>
>>>
>>> or something like that. My details here are probably off by 1 or something.
>>>
>>>
>>> DFDL's Xpath-like expression language has the operators +, -, *, div, idiv, mod, and the math.pow() function. Alas there's no shift nor a bit-wise & and | operator. Generally it is possible to figure out how to achieve what you want by way of mathematical operations.
>>>
>>>
>>> Also note: If you want Link16 specific support we have a channel for FOUO email support which is daffodil-fouo-support@tresys.com.
>>>
>>>
>>> ...mike beckerle
>>>
>>> Tresys
>>>
>>>
>>>
>>> ________________________________
>>> From: Marshall Miller <ma...@quarksecurity.com>
>>> Sent: Thursday, January 11, 2018 2:34:46 PM
>>> To: dev@daffodil.apache.org
>>> Subject: Help implementing schema for SISO-J
>>>
>>> Hello,
>>>
>>> I am trying to implement a schema to describe a LINK-16 message
>>> embedded within a DIS Signal PDU as described by SISO-STD-002 [1],
>>> paragraph 5.2.2.9 and table 5.2.3; commonly referred to as SISO-J.
>>> The problem is that fields are packed into 32 bit words with big
>>> endian byte order, and little endian bitfield order. It is currently
>>> an error to specify a byteOrder of bigEndian with a bitOrder of
>>> leastSignificantBitFirst. Also, I have not found a way to set word
>>> size; it appears that word size is implicitly set to the size of each
>>> field.
>>>
>>> Following is a simple example to demonstrate how fields need to be
>>> interpreted. It is just an example and does not represent any part of
>>> a SISO-J message.
>>>
>>> Data is comprised of 3 unsigned integer bitfields:
>>> slot1 is 12 bits
>>> slot2 is 32 bits
>>> slot3 is 20 bits
>>>
>>> Given an input with the following bytes in hex:
>>> 01 23 45 67 89 AB CD EF
>>>
>>> Parsed results need to be (in hex):
>>> slot1: 0567
>>> slot2: DEF01234
>>> slot3: 089ABC
>>>
>>> Explanation of results:
>>>
>>> Since the bitfields need to consume data in a leastSignificantBitFirst
>>> order, it is easier to represent the input in right-to-left order
>>> where bits are consumed starting at the right and moving toward the
>>> left. Since words are 4 bytes, and in big endian order, the lowest
>>> bit in the 4th byte is the first bit to be consumed. Reordering the
>>> input data and consuming data from right to left yields the correct
>>> field boundaries.
>>>
>>> 89 AB CD EF 01 23 45 67 parse fields right to left
>>> 05 06 07 08 01 02 03 04 input byte position
>>>
>>> slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles,
>>> and slot3 consumes the left 5 nibbles.
>>>
>>> Could you please let me know if there is a way to parse fields in this manner?
>>>
>>>
>>>
>>> Marshall Miller
>>>
>>>
>>> [1] https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=43048&PortalId=0&TabId=105
Re: Help implementing schema for SISO-J
Posted by Mike Beckerle <mb...@tresys.com>.
The byte-order swapping sounds right Marshall.
One of the features for DFDL/Daffodil for which there is much demand is a feature that enables pre-processing passes like what you are describing. Things like base64 encoding, line-folding, byte-order/word swapping, decompress, are all things people need to be able to do on the underlying data stream; not necessarily of the entire data stream, but on just the section of the data stream holding the representation of a specific element within a greater format. E.g., VMF messages are often carried as a compressed payload within a MIL-STD-2045 envelope. Internet messages have attachments that are base64 encoded commonly, etc etc.
In your case, this word/byte swapping would be applicable to the SISO-J headers, but would it apply to the Link16 data being carried by it? If it applies only to the SISO-J packaging part, then you'd need this feature.
The current proposal for this feature (which I'm calling the "layering" or "base64" feature) was actually circulated on this mailing list last week for initial feedback. Implementing this feature in Daffodil is within our 6 month roadmap/plan.
I like this better than trying to come up with an endless proliferation of word, byte, and bit order properties, the interactions of which are not very orthogonal.
As for the way this or any other feature gets incorporated into the DFDL language spec. Basically the cycle is 1) propose and discuss feature 2) prototype feature in Daffodil 3) get experience with feature 4) officially propose for inclusion in DFDL. That's if you are working with Daffodil. If you worked for IBM you would perhaps prototype in their implementation.
If we implement base64 and/or compress maybe we could get you to develop the word-swapping pre/post processor which would would work analogously in Daffodil?
________________________________
From: Marshall Miller <ma...@quarksecurity.com>
Sent: Thursday, January 11, 2018 5:18:26 PM
To: dev@daffodil.apache.org
Subject: Re: Help implementing schema for SISO-J
Also, it occurred to me that it might be possible to perform a stream
transformation on the data before it is parsed. If each 32 bit word
were byte-swapped before parsing fields, then I believe the fields
could be properly parsed in littleEndian byte order with LSBF
bitOrder.
On Thu, Jan 11, 2018 at 4:50 PM, Marshall Miller
<ma...@quarksecurity.com> wrote:
> Mike,
>
> Thanks for the helpful and quick response. I was worried we might
> have to extract the fields with mathematical operations as you
> described.
>
> Hopefully the DFDL Workgroup will add support for this type of packing
> so that LINK-16 schemas can be reused in SISO-J messages. When does
> the workgroup meet next? What do you think the likelihood is for a
> change like this to be accepted?
>
>
> Marshall Miller
>
> On Thu, Jan 11, 2018 at 3:48 PM, Mike Beckerle <mb...@tresys.com> wrote:
>> Marshall,
>>
>>
>> Looks like we'll have to give some thought to whether bigEndian and LSBF can go together. We've had no actual examples of this until now. I will bring that up with the OGF DFDL Workgroup.
>>
>>
>> But to deal with this today, these kinds of mixed bit order situations generally require use of the dfdl:inputValueCalc and dfdl:outputValueCalc properties.
>>
>>
>> A simple example is here:
>>
>>
>> https://github.com/rchampag/dfdl/blob/master/arinc429-label/localizer-deviation-label.dfdl.xsd
>>
>>
>> This was for individual bits, but in your case they would be wider integers.
>>
>>
>> For your 3 fields in the example I suggest you parse as 4 fields
>>
>>
>> h_slot2LSBs - 20 bits
>>
>> h_slot1 - 12 bits
>>
>> h_slot3 - 20 bits
>>
>> h_slot2MSBs - 12 bits
>>
>>
>> You would do this all bigEndian and mostSignificantBitFirst.
>>
>>
>> The above would go into a hidden group so they wouldn't show up in the logical data.
>>
>>
>> Then you would have 3 logical ordinary, not hidden elements
>>
>> slot1 dfdl:inputValueCalc='{ ../h_slot1 }'
>>
>> slot2 dfdl:inputValueCalc='{ ( ../h_slot2MSBs * dfdl:int('x80000' ) + ../h_slot2LSBs }'
>>
>> slot3 dfdl:inputValueCalc='{ ../h_slot3 }'
>>
>>
>> dfdl:int('x80000') constructs a integer from a hex string, and I think that's the right value. Or maybe it's 'x100000'.
>>
>>
>> You can also use math.pow function as in
>>
>>
>> xs:int(math:pow(2, 19))
>>
>>
>> (you need to add this namespace to your xs:schema element xmlns:math='http://www.w3.org/2005/xpath-functions/math')
>>
>>
>> The tricky part is that for unparsing you must invert this using dfdl:outputValueCalc. So the 4 hidden elements would be like this:
>>
>>
>> h_slot2LSBs dfdl:outputValueCalc="{ ../slot2 mod dfdl:int('x80000') }"
>>
>> h_slot1 dfdl:outputValueCalc='{ ../slot1 }'
>>
>> h_slot3 dfdl:outputValueCalc='{ ../slot3 }'
>>
>> h_slot2MSBs dfdl:outputValueCalc="{ ../slot2 idiv dfdl:int('x80000') }"
>>
>>
>> or something like that. My details here are probably off by 1 or something.
>>
>>
>> DFDL's Xpath-like expression language has the operators +, -, *, div, idiv, mod, and the math.pow() function. Alas there's no shift nor a bit-wise & and | operator. Generally it is possible to figure out how to achieve what you want by way of mathematical operations.
>>
>>
>> Also note: If you want Link16 specific support we have a channel for FOUO email support which is daffodil-fouo-support@tresys.com.
>>
>>
>> ...mike beckerle
>>
>> Tresys
>>
>>
>>
>> ________________________________
>> From: Marshall Miller <ma...@quarksecurity.com>
>> Sent: Thursday, January 11, 2018 2:34:46 PM
>> To: dev@daffodil.apache.org
>> Subject: Help implementing schema for SISO-J
>>
>> Hello,
>>
>> I am trying to implement a schema to describe a LINK-16 message
>> embedded within a DIS Signal PDU as described by SISO-STD-002 [1],
>> paragraph 5.2.2.9 and table 5.2.3; commonly referred to as SISO-J.
>> The problem is that fields are packed into 32 bit words with big
>> endian byte order, and little endian bitfield order. It is currently
>> an error to specify a byteOrder of bigEndian with a bitOrder of
>> leastSignificantBitFirst. Also, I have not found a way to set word
>> size; it appears that word size is implicitly set to the size of each
>> field.
>>
>> Following is a simple example to demonstrate how fields need to be
>> interpreted. It is just an example and does not represent any part of
>> a SISO-J message.
>>
>> Data is comprised of 3 unsigned integer bitfields:
>> slot1 is 12 bits
>> slot2 is 32 bits
>> slot3 is 20 bits
>>
>> Given an input with the following bytes in hex:
>> 01 23 45 67 89 AB CD EF
>>
>> Parsed results need to be (in hex):
>> slot1: 0567
>> slot2: DEF01234
>> slot3: 089ABC
>>
>> Explanation of results:
>>
>> Since the bitfields need to consume data in a leastSignificantBitFirst
>> order, it is easier to represent the input in right-to-left order
>> where bits are consumed starting at the right and moving toward the
>> left. Since words are 4 bytes, and in big endian order, the lowest
>> bit in the 4th byte is the first bit to be consumed. Reordering the
>> input data and consuming data from right to left yields the correct
>> field boundaries.
>>
>> 89 AB CD EF 01 23 45 67 parse fields right to left
>> 05 06 07 08 01 02 03 04 input byte position
>>
>> slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles,
>> and slot3 consumes the left 5 nibbles.
>>
>> Could you please let me know if there is a way to parse fields in this manner?
>>
>>
>>
>> Marshall Miller
>>
>>
>> [1] https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=43048&PortalId=0&TabId=105
Re: Help implementing schema for SISO-J
Posted by Marshall Miller <ma...@quarksecurity.com>.
Also, it occurred to me that it might be possible to perform a stream
transformation on the data before it is parsed. If each 32 bit word
were byte-swapped before parsing fields, then I believe the fields
could be properly parsed in littleEndian byte order with LSBF
bitOrder.
On Thu, Jan 11, 2018 at 4:50 PM, Marshall Miller
<ma...@quarksecurity.com> wrote:
> Mike,
>
> Thanks for the helpful and quick response. I was worried we might
> have to extract the fields with mathematical operations as you
> described.
>
> Hopefully the DFDL Workgroup will add support for this type of packing
> so that LINK-16 schemas can be reused in SISO-J messages. When does
> the workgroup meet next? What do you think the likelihood is for a
> change like this to be accepted?
>
>
> Marshall Miller
>
> On Thu, Jan 11, 2018 at 3:48 PM, Mike Beckerle <mb...@tresys.com> wrote:
>> Marshall,
>>
>>
>> Looks like we'll have to give some thought to whether bigEndian and LSBF can go together. We've had no actual examples of this until now. I will bring that up with the OGF DFDL Workgroup.
>>
>>
>> But to deal with this today, these kinds of mixed bit order situations generally require use of the dfdl:inputValueCalc and dfdl:outputValueCalc properties.
>>
>>
>> A simple example is here:
>>
>>
>> https://github.com/rchampag/dfdl/blob/master/arinc429-label/localizer-deviation-label.dfdl.xsd
>>
>>
>> This was for individual bits, but in your case they would be wider integers.
>>
>>
>> For your 3 fields in the example I suggest you parse as 4 fields
>>
>>
>> h_slot2LSBs - 20 bits
>>
>> h_slot1 - 12 bits
>>
>> h_slot3 - 20 bits
>>
>> h_slot2MSBs - 12 bits
>>
>>
>> You would do this all bigEndian and mostSignificantBitFirst.
>>
>>
>> The above would go into a hidden group so they wouldn't show up in the logical data.
>>
>>
>> Then you would have 3 logical ordinary, not hidden elements
>>
>> slot1 dfdl:inputValueCalc='{ ../h_slot1 }'
>>
>> slot2 dfdl:inputValueCalc='{ ( ../h_slot2MSBs * dfdl:int('x80000' ) + ../h_slot2LSBs }'
>>
>> slot3 dfdl:inputValueCalc='{ ../h_slot3 }'
>>
>>
>> dfdl:int('x80000') constructs a integer from a hex string, and I think that's the right value. Or maybe it's 'x100000'.
>>
>>
>> You can also use math.pow function as in
>>
>>
>> xs:int(math:pow(2, 19))
>>
>>
>> (you need to add this namespace to your xs:schema element xmlns:math='http://www.w3.org/2005/xpath-functions/math')
>>
>>
>> The tricky part is that for unparsing you must invert this using dfdl:outputValueCalc. So the 4 hidden elements would be like this:
>>
>>
>> h_slot2LSBs dfdl:outputValueCalc="{ ../slot2 mod dfdl:int('x80000') }"
>>
>> h_slot1 dfdl:outputValueCalc='{ ../slot1 }'
>>
>> h_slot3 dfdl:outputValueCalc='{ ../slot3 }'
>>
>> h_slot2MSBs dfdl:outputValueCalc="{ ../slot2 idiv dfdl:int('x80000') }"
>>
>>
>> or something like that. My details here are probably off by 1 or something.
>>
>>
>> DFDL's Xpath-like expression language has the operators +, -, *, div, idiv, mod, and the math.pow() function. Alas there's no shift nor a bit-wise & and | operator. Generally it is possible to figure out how to achieve what you want by way of mathematical operations.
>>
>>
>> Also note: If you want Link16 specific support we have a channel for FOUO email support which is daffodil-fouo-support@tresys.com.
>>
>>
>> ...mike beckerle
>>
>> Tresys
>>
>>
>>
>> ________________________________
>> From: Marshall Miller <ma...@quarksecurity.com>
>> Sent: Thursday, January 11, 2018 2:34:46 PM
>> To: dev@daffodil.apache.org
>> Subject: Help implementing schema for SISO-J
>>
>> Hello,
>>
>> I am trying to implement a schema to describe a LINK-16 message
>> embedded within a DIS Signal PDU as described by SISO-STD-002 [1],
>> paragraph 5.2.2.9 and table 5.2.3; commonly referred to as SISO-J.
>> The problem is that fields are packed into 32 bit words with big
>> endian byte order, and little endian bitfield order. It is currently
>> an error to specify a byteOrder of bigEndian with a bitOrder of
>> leastSignificantBitFirst. Also, I have not found a way to set word
>> size; it appears that word size is implicitly set to the size of each
>> field.
>>
>> Following is a simple example to demonstrate how fields need to be
>> interpreted. It is just an example and does not represent any part of
>> a SISO-J message.
>>
>> Data is comprised of 3 unsigned integer bitfields:
>> slot1 is 12 bits
>> slot2 is 32 bits
>> slot3 is 20 bits
>>
>> Given an input with the following bytes in hex:
>> 01 23 45 67 89 AB CD EF
>>
>> Parsed results need to be (in hex):
>> slot1: 0567
>> slot2: DEF01234
>> slot3: 089ABC
>>
>> Explanation of results:
>>
>> Since the bitfields need to consume data in a leastSignificantBitFirst
>> order, it is easier to represent the input in right-to-left order
>> where bits are consumed starting at the right and moving toward the
>> left. Since words are 4 bytes, and in big endian order, the lowest
>> bit in the 4th byte is the first bit to be consumed. Reordering the
>> input data and consuming data from right to left yields the correct
>> field boundaries.
>>
>> 89 AB CD EF 01 23 45 67 parse fields right to left
>> 05 06 07 08 01 02 03 04 input byte position
>>
>> slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles,
>> and slot3 consumes the left 5 nibbles.
>>
>> Could you please let me know if there is a way to parse fields in this manner?
>>
>>
>>
>> Marshall Miller
>>
>>
>> [1] https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=43048&PortalId=0&TabId=105
Re: Help implementing schema for SISO-J
Posted by Marshall Miller <ma...@quarksecurity.com>.
Mike,
Thanks for the helpful and quick response. I was worried we might
have to extract the fields with mathematical operations as you
described.
Hopefully the DFDL Workgroup will add support for this type of packing
so that LINK-16 schemas can be reused in SISO-J messages. When does
the workgroup meet next? What do you think the likelihood is for a
change like this to be accepted?
Marshall Miller
On Thu, Jan 11, 2018 at 3:48 PM, Mike Beckerle <mb...@tresys.com> wrote:
> Marshall,
>
>
> Looks like we'll have to give some thought to whether bigEndian and LSBF can go together. We've had no actual examples of this until now. I will bring that up with the OGF DFDL Workgroup.
>
>
> But to deal with this today, these kinds of mixed bit order situations generally require use of the dfdl:inputValueCalc and dfdl:outputValueCalc properties.
>
>
> A simple example is here:
>
>
> https://github.com/rchampag/dfdl/blob/master/arinc429-label/localizer-deviation-label.dfdl.xsd
>
>
> This was for individual bits, but in your case they would be wider integers.
>
>
> For your 3 fields in the example I suggest you parse as 4 fields
>
>
> h_slot2LSBs - 20 bits
>
> h_slot1 - 12 bits
>
> h_slot3 - 20 bits
>
> h_slot2MSBs - 12 bits
>
>
> You would do this all bigEndian and mostSignificantBitFirst.
>
>
> The above would go into a hidden group so they wouldn't show up in the logical data.
>
>
> Then you would have 3 logical ordinary, not hidden elements
>
> slot1 dfdl:inputValueCalc='{ ../h_slot1 }'
>
> slot2 dfdl:inputValueCalc='{ ( ../h_slot2MSBs * dfdl:int('x80000' ) + ../h_slot2LSBs }'
>
> slot3 dfdl:inputValueCalc='{ ../h_slot3 }'
>
>
> dfdl:int('x80000') constructs a integer from a hex string, and I think that's the right value. Or maybe it's 'x100000'.
>
>
> You can also use math.pow function as in
>
>
> xs:int(math:pow(2, 19))
>
>
> (you need to add this namespace to your xs:schema element xmlns:math='http://www.w3.org/2005/xpath-functions/math')
>
>
> The tricky part is that for unparsing you must invert this using dfdl:outputValueCalc. So the 4 hidden elements would be like this:
>
>
> h_slot2LSBs dfdl:outputValueCalc="{ ../slot2 mod dfdl:int('x80000') }"
>
> h_slot1 dfdl:outputValueCalc='{ ../slot1 }'
>
> h_slot3 dfdl:outputValueCalc='{ ../slot3 }'
>
> h_slot2MSBs dfdl:outputValueCalc="{ ../slot2 idiv dfdl:int('x80000') }"
>
>
> or something like that. My details here are probably off by 1 or something.
>
>
> DFDL's Xpath-like expression language has the operators +, -, *, div, idiv, mod, and the math.pow() function. Alas there's no shift nor a bit-wise & and | operator. Generally it is possible to figure out how to achieve what you want by way of mathematical operations.
>
>
> Also note: If you want Link16 specific support we have a channel for FOUO email support which is daffodil-fouo-support@tresys.com.
>
>
> ...mike beckerle
>
> Tresys
>
>
>
> ________________________________
> From: Marshall Miller <ma...@quarksecurity.com>
> Sent: Thursday, January 11, 2018 2:34:46 PM
> To: dev@daffodil.apache.org
> Subject: Help implementing schema for SISO-J
>
> Hello,
>
> I am trying to implement a schema to describe a LINK-16 message
> embedded within a DIS Signal PDU as described by SISO-STD-002 [1],
> paragraph 5.2.2.9 and table 5.2.3; commonly referred to as SISO-J.
> The problem is that fields are packed into 32 bit words with big
> endian byte order, and little endian bitfield order. It is currently
> an error to specify a byteOrder of bigEndian with a bitOrder of
> leastSignificantBitFirst. Also, I have not found a way to set word
> size; it appears that word size is implicitly set to the size of each
> field.
>
> Following is a simple example to demonstrate how fields need to be
> interpreted. It is just an example and does not represent any part of
> a SISO-J message.
>
> Data is comprised of 3 unsigned integer bitfields:
> slot1 is 12 bits
> slot2 is 32 bits
> slot3 is 20 bits
>
> Given an input with the following bytes in hex:
> 01 23 45 67 89 AB CD EF
>
> Parsed results need to be (in hex):
> slot1: 0567
> slot2: DEF01234
> slot3: 089ABC
>
> Explanation of results:
>
> Since the bitfields need to consume data in a leastSignificantBitFirst
> order, it is easier to represent the input in right-to-left order
> where bits are consumed starting at the right and moving toward the
> left. Since words are 4 bytes, and in big endian order, the lowest
> bit in the 4th byte is the first bit to be consumed. Reordering the
> input data and consuming data from right to left yields the correct
> field boundaries.
>
> 89 AB CD EF 01 23 45 67 parse fields right to left
> 05 06 07 08 01 02 03 04 input byte position
>
> slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles,
> and slot3 consumes the left 5 nibbles.
>
> Could you please let me know if there is a way to parse fields in this manner?
>
>
>
> Marshall Miller
>
>
> [1] https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=43048&PortalId=0&TabId=105
Re: Help implementing schema for SISO-J
Posted by Mike Beckerle <mb...@tresys.com>.
Marshall,
Looks like we'll have to give some thought to whether bigEndian and LSBF can go together. We've had no actual examples of this until now. I will bring that up with the OGF DFDL Workgroup.
But to deal with this today, these kinds of mixed bit order situations generally require use of the dfdl:inputValueCalc and dfdl:outputValueCalc properties.
A simple example is here:
https://github.com/rchampag/dfdl/blob/master/arinc429-label/localizer-deviation-label.dfdl.xsd
This was for individual bits, but in your case they would be wider integers.
For your 3 fields in the example I suggest you parse as 4 fields
h_slot2LSBs - 20 bits
h_slot1 - 12 bits
h_slot3 - 20 bits
h_slot2MSBs - 12 bits
You would do this all bigEndian and mostSignificantBitFirst.
The above would go into a hidden group so they wouldn't show up in the logical data.
Then you would have 3 logical ordinary, not hidden elements
slot1 dfdl:inputValueCalc='{ ../h_slot1 }'
slot2 dfdl:inputValueCalc='{ ( ../h_slot2MSBs * dfdl:int('x80000' ) + ../h_slot2LSBs }'
slot3 dfdl:inputValueCalc='{ ../h_slot3 }'
dfdl:int('x80000') constructs a integer from a hex string, and I think that's the right value. Or maybe it's 'x100000'.
You can also use math.pow function as in
xs:int(math:pow(2, 19))
(you need to add this namespace to your xs:schema element xmlns:math='http://www.w3.org/2005/xpath-functions/math')
The tricky part is that for unparsing you must invert this using dfdl:outputValueCalc. So the 4 hidden elements would be like this:
h_slot2LSBs dfdl:outputValueCalc="{ ../slot2 mod dfdl:int('x80000') }"
h_slot1 dfdl:outputValueCalc='{ ../slot1 }'
h_slot3 dfdl:outputValueCalc='{ ../slot3 }'
h_slot2MSBs dfdl:outputValueCalc="{ ../slot2 idiv dfdl:int('x80000') }"
or something like that. My details here are probably off by 1 or something.
DFDL's Xpath-like expression language has the operators +, -, *, div, idiv, mod, and the math.pow() function. Alas there's no shift nor a bit-wise & and | operator. Generally it is possible to figure out how to achieve what you want by way of mathematical operations.
Also note: If you want Link16 specific support we have a channel for FOUO email support which is daffodil-fouo-support@tresys.com.
...mike beckerle
Tresys
________________________________
From: Marshall Miller <ma...@quarksecurity.com>
Sent: Thursday, January 11, 2018 2:34:46 PM
To: dev@daffodil.apache.org
Subject: Help implementing schema for SISO-J
Hello,
I am trying to implement a schema to describe a LINK-16 message
embedded within a DIS Signal PDU as described by SISO-STD-002 [1],
paragraph 5.2.2.9 and table 5.2.3; commonly referred to as SISO-J.
The problem is that fields are packed into 32 bit words with big
endian byte order, and little endian bitfield order. It is currently
an error to specify a byteOrder of bigEndian with a bitOrder of
leastSignificantBitFirst. Also, I have not found a way to set word
size; it appears that word size is implicitly set to the size of each
field.
Following is a simple example to demonstrate how fields need to be
interpreted. It is just an example and does not represent any part of
a SISO-J message.
Data is comprised of 3 unsigned integer bitfields:
slot1 is 12 bits
slot2 is 32 bits
slot3 is 20 bits
Given an input with the following bytes in hex:
01 23 45 67 89 AB CD EF
Parsed results need to be (in hex):
slot1: 0567
slot2: DEF01234
slot3: 089ABC
Explanation of results:
Since the bitfields need to consume data in a leastSignificantBitFirst
order, it is easier to represent the input in right-to-left order
where bits are consumed starting at the right and moving toward the
left. Since words are 4 bytes, and in big endian order, the lowest
bit in the 4th byte is the first bit to be consumed. Reordering the
input data and consuming data from right to left yields the correct
field boundaries.
89 AB CD EF 01 23 45 67 parse fields right to left
05 06 07 08 01 02 03 04 input byte position
slot1 consumes the right 3 nibbles, slot2 consumes the next 8 nibbles,
and slot3 consumes the left 5 nibbles.
Could you please let me know if there is a way to parse fields in this manner?
Marshall Miller
[1] https://www.sisostds.org/DesktopModules/Bring2mind/DMX/Download.aspx?Command=Core_Download&EntryId=43048&PortalId=0&TabId=105