Byte representation

[Sebastian Rühl <se...@googlemail.com.INVALID>]

Hi together,

Working on xml serialization (and boxed [more about that in my next summary mail]) I noticed that we are missing support for byte in mspec:

By (logical) definition a byte is a container of a bits without a immediate semantical meaning. This could be for example a image, binary data of a compiled program etc.
Opposed to a strongly typed definition of uint8 where the meaning would be „a unsigned integer with 8-bit storage“ which could be a sensory data.
When using these in an array I would render the first one (byte) as a hex string (or base64/cdata) “<data>AFFE</data>"  in xml whereas the latter one (uint 8) I would render as „<data><value>175</value><value>254</value></data>. Same would apply to my boxes where I would render the first one as a pretty hex-dump (with line numbers and string representation) and the latter as pretty value boxes with contained integers.

For that we would add a write(byte[]) read(byte[]) to the Writer/Reader and call them from the template whenever we are sure that the data we write is of the definition „byte“. Same could be done to single values and their rendering of course.

For the api-user and the data itself on the wire nothing would change, it would just make testing and debugging a whole lot easier.

What do you think?

Sebastian

Re: Byte representation

[Łukasz Dywicki <lu...@code-house.org>]

Its Java so its made in usual Java way which involves definition of
specification (https://jcp.org/en/jsr/detail?id=353) which tries to
define common ground for all JSON processors... so if you stick with it
in theory you could get multiple libs on it.
In practice I don't think gson took the JSR353 idea. Another way to map
XML is XStream which is a bit ancient but I think a bit easier to
customize than Jackson.

Anyhow, whenever tool we take we'll be able to adjust. I wouldn't worry
much of JSR353. One additional point - not sure if you spotted it but if
you use IntelliJ and have a failure in driver suite test then double
click on test will bring you to step which failed.
There are some left overs to make it navigate to failed assertion:
https://github.com/junit-team/junit5/issues/2504

Cheers,
Łukasz

On 21.04.2021 11:04, Sebastian Rühl wrote:
> Small correction: not having native json support in java (is this 2021? :D) in might be best to still use Jackson, but in a more native way where we don’t work with annotations for the generated classes
> 
> Sebastian
> 
>> Am 21.04.2021 um 10:36 schrieb Sebastian Rühl <se...@googlemail.com>:
>>
>> Talking about Jackson, this can be soon be replaced with our native generated implementation.
>> In Golang xml/json is already ported, in java only xml for now.
>> When I’m done with that we have full control over the serialization which then is language agnostic and dictated by what is defined in mspec. 
>> (e.g. if mspec defines that the element is byte[] or byte then it will be generated as such (hex) across every language. For debugging purposes you can look at my box rendering which contains every representation [hex, natural, possible string on hex dump]. At the moment I haven’t ported it to java yet)
>>
>> Sebastian
>>
>>> Am 21.04.2021 um 10:27 schrieb Łukasz Dywicki <lu...@code-house.org>:
>>>
>>> Looking at very core of the issue with byte type representation (at
>>> least in XML suits) I believe it was affected by mapping to/from Java.
>>> Jackson has its own ways handling such payloads.
>>> We really should map suites according to mspec and keep values in tests
>>> according to their "canonical" representation. Without pinning it to any
>>> particular language.
>>> As far I remember we currently map byte[] payloads to hex strings, but
>>> sometimes such representation is helpful also for digits. Earlier we had
>>> base64 for byte arrays which did make it pretty hard to debug tests. If
>>> we could map single bytes to hex that also would be great cause turning
>>> it into digit by Jackson is not very helpful.
>>>
>>> Best,
>>> Łukasz
>>
> 

Re: Byte representation

[Sebastian Rühl <se...@googlemail.com.INVALID>]

Small correction: not having native json support in java (is this 2021? :D) in might be best to still use Jackson, but in a more native way where we don’t work with annotations for the generated classes

Sebastian

> Am 21.04.2021 um 10:36 schrieb Sebastian Rühl <se...@googlemail.com>:
> 
> Talking about Jackson, this can be soon be replaced with our native generated implementation.
> In Golang xml/json is already ported, in java only xml for now.
> When I’m done with that we have full control over the serialization which then is language agnostic and dictated by what is defined in mspec. 
> (e.g. if mspec defines that the element is byte[] or byte then it will be generated as such (hex) across every language. For debugging purposes you can look at my box rendering which contains every representation [hex, natural, possible string on hex dump]. At the moment I haven’t ported it to java yet)
> 
> Sebastian
> 
>> Am 21.04.2021 um 10:27 schrieb Łukasz Dywicki <lu...@code-house.org>:
>> 
>> Looking at very core of the issue with byte type representation (at
>> least in XML suits) I believe it was affected by mapping to/from Java.
>> Jackson has its own ways handling such payloads.
>> We really should map suites according to mspec and keep values in tests
>> according to their "canonical" representation. Without pinning it to any
>> particular language.
>> As far I remember we currently map byte[] payloads to hex strings, but
>> sometimes such representation is helpful also for digits. Earlier we had
>> base64 for byte arrays which did make it pretty hard to debug tests. If
>> we could map single bytes to hex that also would be great cause turning
>> it into digit by Jackson is not very helpful.
>> 
>> Best,
>> Łukasz
> 

Re: Byte representation

[Sebastian Rühl <se...@googlemail.com.INVALID>]

Talking about Jackson, this can be soon be replaced with our native generated implementation.
In Golang xml/json is already ported, in java only xml for now.
When I’m done with that we have full control over the serialization which then is language agnostic and dictated by what is defined in mspec. 
(e.g. if mspec defines that the element is byte[] or byte then it will be generated as such (hex) across every language. For debugging purposes you can look at my box rendering which contains every representation [hex, natural, possible string on hex dump]. At the moment I haven’t ported it to java yet)

Sebastian

> Am 21.04.2021 um 10:27 schrieb Łukasz Dywicki <lu...@code-house.org>:
> 
> Looking at very core of the issue with byte type representation (at
> least in XML suits) I believe it was affected by mapping to/from Java.
> Jackson has its own ways handling such payloads.
> We really should map suites according to mspec and keep values in tests
> according to their "canonical" representation. Without pinning it to any
> particular language.
> As far I remember we currently map byte[] payloads to hex strings, but
> sometimes such representation is helpful also for digits. Earlier we had
> base64 for byte arrays which did make it pretty hard to debug tests. If
> we could map single bytes to hex that also would be great cause turning
> it into digit by Jackson is not very helpful.
> 
> Best,
> Łukasz

Re: AW: Byte representation

[Łukasz Dywicki <lu...@code-house.org>]

Looking at very core of the issue with byte type representation (at
least in XML suits) I believe it was affected by mapping to/from Java.
Jackson has its own ways handling such payloads.
We really should map suites according to mspec and keep values in tests
according to their "canonical" representation. Without pinning it to any
particular language.
As far I remember we currently map byte[] payloads to hex strings, but
sometimes such representation is helpful also for digits. Earlier we had
base64 for byte arrays which did make it pretty hard to debug tests. If
we could map single bytes to hex that also would be great cause turning
it into digit by Jackson is not very helpful.

Best,
Łukasz

On 20.04.2021 18:00, Christofer Dutz wrote:
> Hi Sebastian,
> 
> Thanks for taking our discussion to the list :)
> 
> I would like to lay emphasis on the fact that "byte" would not have any size ... it is defined as fixed 8-bit element so it should have been:
> "By (logical) definition a byte is a container of exactly 8 bits without an immediate semantical meaning."
> 
> I know that in the past I objected the concept of a "byte" in mspec. But let me clarify ... I still do if we would continue to provide a length.
> But I do (now) see big benefits in a "byte" without size (So more like a "bit" which's length is inherently defined by its name)
> 
> In java the natural datatype for data is byte, however it's a signed 8-bit value. By using an array of "uint 8" we internally create an array of short values.
> If we used an mspec type of "byte" each language could decide which is their natural type for these values. In java this would be "byte" as "int 8" ... in Go it would be a "uint 8" same as in C.
> 
> And to be complete, you should probably also add a: write(byte) read(byte) too.
> 
> If we say "byte" is always 8 but, I would be in strong favor of this change. If not I would be in strong opposition ;-)
> 
> Chris
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Sebastian Rühl <se...@googlemail.com.INVALID> 
> Gesendet: Dienstag, 20. April 2021 17:28
> An: dev@plc4x.apache.org
> Betreff: Byte representation
> 
> Hi together,
> 
> Working on xml serialization (and boxed [more about that in my next summary mail]) I noticed that we are missing support for byte in mspec:
> 
> By (logical) definition a byte is a container of a bits without a immediate semantical meaning. This could be for example a image, binary data of a compiled program etc.
> Opposed to a strongly typed definition of uint8 where the meaning would be „a unsigned integer with 8-bit storage“ which could be a sensory data.
> When using these in an array I would render the first one (byte) as a hex string (or base64/cdata) “<data>AFFE</data>"  in xml whereas the latter one (uint 8) I would render as „<data><value>175</value><value>254</value></data>. Same would apply to my boxes where I would render the first one as a pretty hex-dump (with line numbers and string representation) and the latter as pretty value boxes with contained integers.
> 
> For that we would add a write(byte[]) read(byte[]) to the Writer/Reader and call them from the template whenever we are sure that the data we write is of the definition „byte“. Same could be done to single values and their rendering of course.
> 
> For the api-user and the data itself on the wire nothing would change, it would just make testing and debugging a whole lot easier.
> 
> What do you think?
> 
> Sebastian
> 

AW: Byte representation

[Christofer Dutz <ch...@c-ware.de>]

Hi Sebastian,

Thanks for taking our discussion to the list :)

I would like to lay emphasis on the fact that "byte" would not have any size ... it is defined as fixed 8-bit element so it should have been:
"By (logical) definition a byte is a container of exactly 8 bits without an immediate semantical meaning."

I know that in the past I objected the concept of a "byte" in mspec. But let me clarify ... I still do if we would continue to provide a length.
But I do (now) see big benefits in a "byte" without size (So more like a "bit" which's length is inherently defined by its name)

In java the natural datatype for data is byte, however it's a signed 8-bit value. By using an array of "uint 8" we internally create an array of short values.
If we used an mspec type of "byte" each language could decide which is their natural type for these values. In java this would be "byte" as "int 8" ... in Go it would be a "uint 8" same as in C.

And to be complete, you should probably also add a: write(byte) read(byte) too.

If we say "byte" is always 8 but, I would be in strong favor of this change. If not I would be in strong opposition ;-)

Chris



-----Ursprüngliche Nachricht-----
Von: Sebastian Rühl <se...@googlemail.com.INVALID> 
Gesendet: Dienstag, 20. April 2021 17:28
An: dev@plc4x.apache.org
Betreff: Byte representation

Hi together,

Working on xml serialization (and boxed [more about that in my next summary mail]) I noticed that we are missing support for byte in mspec:

By (logical) definition a byte is a container of a bits without a immediate semantical meaning. This could be for example a image, binary data of a compiled program etc.
Opposed to a strongly typed definition of uint8 where the meaning would be „a unsigned integer with 8-bit storage“ which could be a sensory data.
When using these in an array I would render the first one (byte) as a hex string (or base64/cdata) “<data>AFFE</data>"  in xml whereas the latter one (uint 8) I would render as „<data><value>175</value><value>254</value></data>. Same would apply to my boxes where I would render the first one as a pretty hex-dump (with line numbers and string representation) and the latter as pretty value boxes with contained integers.

For that we would add a write(byte[]) read(byte[]) to the Writer/Reader and call them from the template whenever we are sure that the data we write is of the definition „byte“. Same could be done to single values and their rendering of course.

For the api-user and the data itself on the wire nothing would change, it would just make testing and debugging a whole lot easier.

What do you think?

Sebastian