Have you ever seen an initiator used in a binary file format?

["Costello, Roger L." <co...@mitre.org>]

Hello DFDL community,

Have you ever seen an initiator used in a binary file format?

If yes, would you describe how it was used, please?

/Roger

Re: Have you ever seen an initiator used in a binary file format?

["Beckerle, Mike" <mb...@tresys.com>]

Definitely there are examples of binary data with delimiters of various kinds. I can't say I've seen initiators specifically, but what gives rise to this is primarily packed decimal computing.

Many mainframes and mini-computers have decimal instruction sets that directly operate math on packed and zoned decimal representations. These machines (ibm 390 family, and ibm AS400 family, and others) are still in heavy use.

Delimiters typically would come up in packed-decimal data, where the bit pattern of valid data is always going to be 0-9 digits + C, D, F trailing sign nibbles

E.g.,

decimal 20161 (in a 5 digit fixed length field) is bytes 00 20 16 1C

decimal -1 similarly would be 00 00 00 1D

unsigned decimal 345 would be 00 00 34 5F

Hence, a byte like 0xFF can be used as a delimiter, as can AA, AB, 2A, and so forth, because they cannot appear in valid data. Byte 2A is an ascii "*" character, so using that as a delimiter is convenient, because viewing data as text might look like gibberish, but the "*" characters are easily picked out as separators.

The DFDL Schema on github for IBM4690TLog format uses delimiters in packed decimal data. This is a format created by some point-of-sale terminals (cash registers).

DFDL supports 3 variants of packed decimal. Basic "packed" is what I just showed above. BCD is only unsigned nibbles, no sign nibbles, and ibm4690Tlog is a variant of regular packed with slightly different conventions. On top of this, the convention of C, D for +/- signs is sometimes different, so there are properties in DFDL to customize this.

It is possible, though I haven't seen it, to do delimited zoned decimal also, again because not all bytes are valid data. I bet it is out there somewhere though.
________________________________
From: Sloane, Brandon <bs...@tresys.com>
Sent: Wednesday, September 18, 2019 10:27 AM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Re: Have you ever seen an initiator used in a binary file format?

The magic number at the start of many binary formats is arguably an initiator for the entire format.

I have also seen binary formats that prefix all data fields with a tag indicating what type of field they are. This tag could be thought of as an initiator to the corresponding field. See DER encoding for an example of this https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One#Example_encoded_in_DER

Having said that, I do not think I would describe this as an initiator without being prompted. I believe DFDL requires that initiators be expressed as character stings, and therefore be a multiple of 8 bits long. For a binary format, this is a rather arbitrary restriction.

________________________________
From: Costello, Roger L. <co...@mitre.org>
Sent: Wednesday, September 18, 2019 8:14 AM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Have you ever seen an initiator used in a binary file format?

Hello DFDL community,

Have you ever seen an initiator used in a binary file format?

If yes, would you describe how it was used, please?

/Roger

Re: Have you ever seen an initiator used in a binary file format?

["Sloane, Brandon" <bs...@tresys.com>]

The magic number at the start of many binary formats is arguably an initiator for the entire format.

I have also seen binary formats that prefix all data fields with a tag indicating what type of field they are. This tag could be thought of as an initiator to the corresponding field. See DER encoding for an example of this https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One#Example_encoded_in_DER

Having said that, I do not think I would describe this as an initiator without being prompted. I believe DFDL requires that initiators be expressed as character stings, and therefore be a multiple of 8 bits long. For a binary format, this is a rather arbitrary restriction.

________________________________
From: Costello, Roger L. <co...@mitre.org>
Sent: Wednesday, September 18, 2019 8:14 AM
To: users@daffodil.apache.org <us...@daffodil.apache.org>
Subject: Have you ever seen an initiator used in a binary file format?

Hello DFDL community,

Have you ever seen an initiator used in a binary file format?

If yes, would you describe how it was used, please?

/Roger