You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by John Wass <jw...@gmail.com> on 2021/11/01 11:17:58 UTC
Hex editor operations
Some thoughts on ways the hex editor would interact with an input stream
and the debugger. I visualize this using a context sensitive menu in the
hex editor as an entrypoint, where these are some of the operations that
could be offered.
1. Add/Delete/Mask/Set
* Individual bytes
* Blocks of bytes
2. Copy/Paste bytes
* Use system clipboard for interoperability
3. Find/Replace bytes
* In a selection
* Across entire file
4. Set breakpoints on bytes
* Stops execution at related point in schema
* Would require custom additions to DAP backend
5. Set run options on bytes
* Debug-to this byte (and break)
* Start from this byte (skipping previous)
The idea with breakpoints and run operations is that they would behave as a
normal breakpoint in code would. I don't know if this is functionality for
the first pass of the editor, but it is definitely something to keep in
mind while designing that first pass.
What other operations could be supported?
Re: Hex editor operations
Posted by John Wass <jw...@gmail.com>.
> I’m not sure if this fits Daffodil’s needs but it could be interesting.
BinEd is Java, we need something in JavaScript.
> Here's an older paper about an innovative hex editor:
There are interesting concepts in there. Some of them, the lenses for
example, are similar to things we have been discussing here.
On Thu, Nov 4, 2021 at 3:58 PM Dave Fisher <wa...@apache.org> wrote:
> There’s a guy with hex editor trying to see if he can make it into a
> community.
>
> https://lists.apache.org/thread/5cbcmfw08002p5ttgyd43kt4vq4c17o8
>
> https://bined.exbin.org/
>
> I’m not sure if this fits Daffodil’s needs but it could be interesting.
>
> Regards,
> Dave
>
> > On Nov 4, 2021, at 12:51 PM, Larry Barber <la...@nteligen.com>
> wrote:
> >
> > Here's an older paper about an innovative hex editor:
> >
> > Abstract
> > The analysis of binary data remains a challenge, especially for large or
> potentially inconsistent files. Traditionally, hex editors only make
> limited use of semantic information available to the user. We present an
> editor that supports user-supplied semantic data definitions. This semantic
> information is used throughout the program to realize semantic data
> visualization and data exploration capabilities not present in similar
> systems. Visualization and human-computer interaction techniques are
> applied. We show that this makes recognizing the structure of unknown or
> inconsistent data much more effective. Our approach demonstrates concepts
> that can be applied to the visual analysis of raw data in general.
> >
> >
> https://www.researchgate.net/publication/220836091_Vide_An_editor_for_the_visual_exploration_of_raw_data
> >
> > -----Original Message-----
> > From: Mike Beckerle <mb...@apache.org>
> > Sent: Monday, November 1, 2021 10:31 PM
> > To: dev@daffodil.apache.org
> > Subject: Re: Hex editor operations
> >
> > This is a good list.
> >
> > One thing I think is important is that often one is dealing with hex
> data, but one needs to consider bit fields that do not respect byte
> boundaries.
> >
> > I suggest we need a way to expand a single byte into a small
> presentation of 8 bits, allowing editing of just those 8 bits individually,
> is of value as part of a hex-editor. Perhaps that is what you meant by
> Mask/Set operations? I think switching to a full-fledged all 1's and 0's
> display mode is only for entirely non-byte-oriented data. Anything
> byte-oriented people users will want to use hex, and occasionally if they
> need to flip bits, a way for them to expand a byte or small run of bytes to
> 1's and 0's, but then collapse back to hex is likely very helpful.
> >
> > But I also suggest creating a minimal hex editor version first, then we
> get experience with it.
> >
> > Any little paper scribble exercises we find ourselves having to do on
> the side of using the editor, those are good candidates for things the UI
> should directly support.
> >
> > E.g. I have written things down on paper or in a text editor like:
> >
> > BF.32.A5.AC.(01|10 1101.101|1 0001)
> >
> > In a real UI these distinctions could be done quite differently.
> > But the distinctions I'm making are dots separating bytes, pipes
> separating bit fields of length 2, 9, and 5 respectively, and parentheses
> indicating bytes expanded out to a bits region for display, e.g., so that
> 01 for 2 bits isn't confused with hex 01 8 bits, and here I have each
> nibble of 4 bits space-separated.
> >
> > Ultimately, we need a graphical means providing:
> > (a) a way of escaping from hex to bits for just a small region of the
> data where you care about the partial byte fields.
> > (b) a way of setting off bit fields from each other, that doesn't
> entirely lose the separation of hex digits.
> >
> > Then if say, that right-most bit field value is incorrect, you should be
> able to set editing focus on a bit and flip it.
> >
> > I would add to the "parse until this byte", or "start from this byte" to
> enable narrowing at both the start and end - parse just these bytes (in an
> identified region by start and end). The use case I have in mind for this
> is sort of like unit testing. You narrow the data to just one part, then
> you specify to parse it not with the root element of the DFDL schema, but
> with a sub-element that you want to test.
> >
> > On Mon, Nov 1, 2021 at 6:19 AM John Wass <jw...@gmail.com> wrote:
> >
> >> Some thoughts on ways the hex editor would interact with an input
> >> stream and the debugger. I visualize this using a context sensitive
> >> menu in the hex editor as an entrypoint, where these are some of the
> >> operations that could be offered.
> >>
> >> 1. Add/Delete/Mask/Set
> >> * Individual bytes
> >> * Blocks of bytes
> >> 2. Copy/Paste bytes
> >> * Use system clipboard for interoperability 3. Find/Replace bytes
> >> * In a selection
> >> * Across entire file
> >> 4. Set breakpoints on bytes
> >> * Stops execution at related point in schema
> >> * Would require custom additions to DAP backend 5. Set run options
> >> on bytes
> >> * Debug-to this byte (and break)
> >> * Start from this byte (skipping previous)
> >>
> >> The idea with breakpoints and run operations is that they would behave
> >> as a normal breakpoint in code would. I don't know if this is
> >> functionality for the first pass of the editor, but it is definitely
> >> something to keep in mind while designing that first pass.
> >>
> >> What other operations could be supported?
> >>
>
>
Re: Hex editor operations
Posted by Dave Fisher <wa...@apache.org>.
There’s a guy with hex editor trying to see if he can make it into a community.
https://lists.apache.org/thread/5cbcmfw08002p5ttgyd43kt4vq4c17o8
https://bined.exbin.org/
I’m not sure if this fits Daffodil’s needs but it could be interesting.
Regards,
Dave
> On Nov 4, 2021, at 12:51 PM, Larry Barber <la...@nteligen.com> wrote:
>
> Here's an older paper about an innovative hex editor:
>
> Abstract
> The analysis of binary data remains a challenge, especially for large or potentially inconsistent files. Traditionally, hex editors only make limited use of semantic information available to the user. We present an editor that supports user-supplied semantic data definitions. This semantic information is used throughout the program to realize semantic data visualization and data exploration capabilities not present in similar systems. Visualization and human-computer interaction techniques are applied. We show that this makes recognizing the structure of unknown or inconsistent data much more effective. Our approach demonstrates concepts that can be applied to the visual analysis of raw data in general.
>
> https://www.researchgate.net/publication/220836091_Vide_An_editor_for_the_visual_exploration_of_raw_data
>
> -----Original Message-----
> From: Mike Beckerle <mb...@apache.org>
> Sent: Monday, November 1, 2021 10:31 PM
> To: dev@daffodil.apache.org
> Subject: Re: Hex editor operations
>
> This is a good list.
>
> One thing I think is important is that often one is dealing with hex data, but one needs to consider bit fields that do not respect byte boundaries.
>
> I suggest we need a way to expand a single byte into a small presentation of 8 bits, allowing editing of just those 8 bits individually, is of value as part of a hex-editor. Perhaps that is what you meant by Mask/Set operations? I think switching to a full-fledged all 1's and 0's display mode is only for entirely non-byte-oriented data. Anything byte-oriented people users will want to use hex, and occasionally if they need to flip bits, a way for them to expand a byte or small run of bytes to 1's and 0's, but then collapse back to hex is likely very helpful.
>
> But I also suggest creating a minimal hex editor version first, then we get experience with it.
>
> Any little paper scribble exercises we find ourselves having to do on the side of using the editor, those are good candidates for things the UI should directly support.
>
> E.g. I have written things down on paper or in a text editor like:
>
> BF.32.A5.AC.(01|10 1101.101|1 0001)
>
> In a real UI these distinctions could be done quite differently.
> But the distinctions I'm making are dots separating bytes, pipes separating bit fields of length 2, 9, and 5 respectively, and parentheses indicating bytes expanded out to a bits region for display, e.g., so that 01 for 2 bits isn't confused with hex 01 8 bits, and here I have each nibble of 4 bits space-separated.
>
> Ultimately, we need a graphical means providing:
> (a) a way of escaping from hex to bits for just a small region of the data where you care about the partial byte fields.
> (b) a way of setting off bit fields from each other, that doesn't entirely lose the separation of hex digits.
>
> Then if say, that right-most bit field value is incorrect, you should be able to set editing focus on a bit and flip it.
>
> I would add to the "parse until this byte", or "start from this byte" to enable narrowing at both the start and end - parse just these bytes (in an identified region by start and end). The use case I have in mind for this is sort of like unit testing. You narrow the data to just one part, then you specify to parse it not with the root element of the DFDL schema, but with a sub-element that you want to test.
>
> On Mon, Nov 1, 2021 at 6:19 AM John Wass <jw...@gmail.com> wrote:
>
>> Some thoughts on ways the hex editor would interact with an input
>> stream and the debugger. I visualize this using a context sensitive
>> menu in the hex editor as an entrypoint, where these are some of the
>> operations that could be offered.
>>
>> 1. Add/Delete/Mask/Set
>> * Individual bytes
>> * Blocks of bytes
>> 2. Copy/Paste bytes
>> * Use system clipboard for interoperability 3. Find/Replace bytes
>> * In a selection
>> * Across entire file
>> 4. Set breakpoints on bytes
>> * Stops execution at related point in schema
>> * Would require custom additions to DAP backend 5. Set run options
>> on bytes
>> * Debug-to this byte (and break)
>> * Start from this byte (skipping previous)
>>
>> The idea with breakpoints and run operations is that they would behave
>> as a normal breakpoint in code would. I don't know if this is
>> functionality for the first pass of the editor, but it is definitely
>> something to keep in mind while designing that first pass.
>>
>> What other operations could be supported?
>>
RE: Hex editor operations
Posted by Larry Barber <la...@nteligen.com>.
Here's an older paper about an innovative hex editor:
Abstract
The analysis of binary data remains a challenge, especially for large or potentially inconsistent files. Traditionally, hex editors only make limited use of semantic information available to the user. We present an editor that supports user-supplied semantic data definitions. This semantic information is used throughout the program to realize semantic data visualization and data exploration capabilities not present in similar systems. Visualization and human-computer interaction techniques are applied. We show that this makes recognizing the structure of unknown or inconsistent data much more effective. Our approach demonstrates concepts that can be applied to the visual analysis of raw data in general.
https://www.researchgate.net/publication/220836091_Vide_An_editor_for_the_visual_exploration_of_raw_data
-----Original Message-----
From: Mike Beckerle <mb...@apache.org>
Sent: Monday, November 1, 2021 10:31 PM
To: dev@daffodil.apache.org
Subject: Re: Hex editor operations
This is a good list.
One thing I think is important is that often one is dealing with hex data, but one needs to consider bit fields that do not respect byte boundaries.
I suggest we need a way to expand a single byte into a small presentation of 8 bits, allowing editing of just those 8 bits individually, is of value as part of a hex-editor. Perhaps that is what you meant by Mask/Set operations? I think switching to a full-fledged all 1's and 0's display mode is only for entirely non-byte-oriented data. Anything byte-oriented people users will want to use hex, and occasionally if they need to flip bits, a way for them to expand a byte or small run of bytes to 1's and 0's, but then collapse back to hex is likely very helpful.
But I also suggest creating a minimal hex editor version first, then we get experience with it.
Any little paper scribble exercises we find ourselves having to do on the side of using the editor, those are good candidates for things the UI should directly support.
E.g. I have written things down on paper or in a text editor like:
BF.32.A5.AC.(01|10 1101.101|1 0001)
In a real UI these distinctions could be done quite differently.
But the distinctions I'm making are dots separating bytes, pipes separating bit fields of length 2, 9, and 5 respectively, and parentheses indicating bytes expanded out to a bits region for display, e.g., so that 01 for 2 bits isn't confused with hex 01 8 bits, and here I have each nibble of 4 bits space-separated.
Ultimately, we need a graphical means providing:
(a) a way of escaping from hex to bits for just a small region of the data where you care about the partial byte fields.
(b) a way of setting off bit fields from each other, that doesn't entirely lose the separation of hex digits.
Then if say, that right-most bit field value is incorrect, you should be able to set editing focus on a bit and flip it.
I would add to the "parse until this byte", or "start from this byte" to enable narrowing at both the start and end - parse just these bytes (in an identified region by start and end). The use case I have in mind for this is sort of like unit testing. You narrow the data to just one part, then you specify to parse it not with the root element of the DFDL schema, but with a sub-element that you want to test.
On Mon, Nov 1, 2021 at 6:19 AM John Wass <jw...@gmail.com> wrote:
> Some thoughts on ways the hex editor would interact with an input
> stream and the debugger. I visualize this using a context sensitive
> menu in the hex editor as an entrypoint, where these are some of the
> operations that could be offered.
>
> 1. Add/Delete/Mask/Set
> * Individual bytes
> * Blocks of bytes
> 2. Copy/Paste bytes
> * Use system clipboard for interoperability 3. Find/Replace bytes
> * In a selection
> * Across entire file
> 4. Set breakpoints on bytes
> * Stops execution at related point in schema
> * Would require custom additions to DAP backend 5. Set run options
> on bytes
> * Debug-to this byte (and break)
> * Start from this byte (skipping previous)
>
> The idea with breakpoints and run operations is that they would behave
> as a normal breakpoint in code would. I don't know if this is
> functionality for the first pass of the editor, but it is definitely
> something to keep in mind while designing that first pass.
>
> What other operations could be supported?
>
Re: Hex editor operations
Posted by John Wass <jw...@gmail.com>.
Thinking about copying/pasting of data... if we can use the system
clipboard it could greatly improve interoperability and usability.
An example scenario would be embedding the contents of a file (A) in
another (B) by copying A from the host with ctrl+c, and then pasting in the
editor, where B is open, using ctrl+v or right click.
This isn't a must have initially but when a clipboard functionality is
implemented, integrating with the system clipboard should be considered.
On Fri, Nov 5, 2021 at 10:10 AM Mike Beckerle <mb...@apache.org> wrote:
> +1 for a properties panel where you can click on a byte, and it shows the
> bits off to the side/corner along with other potentially useful stuff: the
> position of the byte in bytes and in bits, it's value as decimal, its value
> as an ascii char, etc. This would take very little screen space.
>
> On Wed, Nov 3, 2021 at 8:01 AM John Wass <jw...@gmail.com> wrote:
>
> > Yep, I agree with all the bits about bits.
> >
> > The display of non-byte delimited data is covered by the concept of a
> > composable UI that allows for rendering ranges within the file
> differently
> > while laying out these ranges in the proper order and in a seamless
> > presentation. The editing of such data would also be straightforward.
> The
> > bit view has a bit editor associated with it, allowing inline
> modification
> > of bits, just like you would expect in the hex editor.
> >
> > Where it gets fuzzy is editing in a representation different from the
> > display, eg. editing bits from a byte view.
> >
> > > a way to expand a single byte into a small presentation
> > > of 8 bits, allowing editing of just those 8 bits individually
> >
> > I can visualize this a few ways, not sure of what is the best for
> > usability. A properties pane type component when a byte is selected
> might
> > be the right one to start with.
> >
> > > Perhaps that is what you meant by Mask/Set operations?
> >
> > I was only considering a byte view; "Mask" was referring to bit
> > manipulation of each byte in a range, "Set" as a verb, simply to update
> the
> > value of an existing byte.
> >
> > Your thoughts about the bit based views fill in the gaps around bits and
> we
> > can consider similar context operations in that view as those I mentioned
> > for the byte view.
> >
> > > I would add to the "parse until this byte", or "start from this byte"
> to
> > > enable narrowing at both the start and end - parse just these bytes
> >
> > Yeah I like this. The interface between hex editor and debugger commands
> > is going to be an important extension point.
> >
> >
> >
> >
> > On Mon, Nov 1, 2021 at 10:31 PM Mike Beckerle <mb...@apache.org>
> > wrote:
> >
> > > This is a good list.
> > >
> > > One thing I think is important is that often one is dealing with hex
> > data,
> > > but one needs to consider bit fields that do not respect byte
> boundaries.
> > >
> > > I suggest we need a way to expand a single byte into a small
> presentation
> > > of 8 bits, allowing editing of just those 8 bits individually, is of
> > value
> > > as part of a hex-editor. Perhaps that is what you meant by Mask/Set
> > > operations? I think switching to a full-fledged all 1's and 0's display
> > > mode is only for entirely non-byte-oriented data. Anything
> byte-oriented
> > > people users will want to use hex, and occasionally if they need to
> flip
> > > bits, a way for them to expand a byte or small run of bytes to 1's and
> > 0's,
> > > but then collapse back to hex is likely very helpful.
> > >
> > > But I also suggest creating a minimal hex editor version first, then we
> > get
> > > experience with it.
> > >
> > > Any little paper scribble exercises we find ourselves having to do on
> the
> > > side of using the editor, those are good candidates for things the UI
> > > should directly support.
> > >
> > > E.g. I have written things down on paper or in a text editor like:
> > >
> > > BF.32.A5.AC.(01|10 1101.101|1 0001)
> > >
> > > In a real UI these distinctions could be done quite differently.
> > > But the distinctions I'm making are dots separating bytes, pipes
> > separating
> > > bit fields of length 2, 9, and 5 respectively, and parentheses
> indicating
> > > bytes expanded out to a bits region for display, e.g., so that 01 for 2
> > > bits isn't confused with hex 01 8 bits, and here I have each nibble of
> 4
> > > bits space-separated.
> > >
> > > Ultimately, we need a graphical means providing:
> > > (a) a way of escaping from hex to bits for just a small region of the
> > data
> > > where you care about the partial byte fields.
> > > (b) a way of setting off bit fields from each other, that doesn't
> > entirely
> > > lose the separation of hex digits.
> > >
> > > Then if say, that right-most bit field value is incorrect, you should
> be
> > > able to set editing focus on a bit and flip it.
> > >
> > > I would add to the "parse until this byte", or "start from this byte"
> to
> > > enable narrowing at both the start and end - parse just these bytes (in
> > an
> > > identified region by start and end). The use case I have in mind for
> > this
> > > is sort of like unit testing. You narrow the data to just one part,
> then
> > > you specify to parse it not with the root element of the DFDL schema,
> but
> > > with a sub-element that you want to test.
> > >
> > > On Mon, Nov 1, 2021 at 6:19 AM John Wass <jw...@gmail.com> wrote:
> > >
> > > > Some thoughts on ways the hex editor would interact with an input
> > stream
> > > > and the debugger. I visualize this using a context sensitive menu in
> > the
> > > > hex editor as an entrypoint, where these are some of the operations
> > that
> > > > could be offered.
> > > >
> > > > 1. Add/Delete/Mask/Set
> > > > * Individual bytes
> > > > * Blocks of bytes
> > > > 2. Copy/Paste bytes
> > > > * Use system clipboard for interoperability
> > > > 3. Find/Replace bytes
> > > > * In a selection
> > > > * Across entire file
> > > > 4. Set breakpoints on bytes
> > > > * Stops execution at related point in schema
> > > > * Would require custom additions to DAP backend
> > > > 5. Set run options on bytes
> > > > * Debug-to this byte (and break)
> > > > * Start from this byte (skipping previous)
> > > >
> > > > The idea with breakpoints and run operations is that they would
> behave
> > > as a
> > > > normal breakpoint in code would. I don't know if this is
> functionality
> > > for
> > > > the first pass of the editor, but it is definitely something to keep
> in
> > > > mind while designing that first pass.
> > > >
> > > > What other operations could be supported?
> > > >
> > >
> >
>
Re: Hex editor operations
Posted by Mike Beckerle <mb...@apache.org>.
+1 for a properties panel where you can click on a byte, and it shows the
bits off to the side/corner along with other potentially useful stuff: the
position of the byte in bytes and in bits, it's value as decimal, its value
as an ascii char, etc. This would take very little screen space.
On Wed, Nov 3, 2021 at 8:01 AM John Wass <jw...@gmail.com> wrote:
> Yep, I agree with all the bits about bits.
>
> The display of non-byte delimited data is covered by the concept of a
> composable UI that allows for rendering ranges within the file differently
> while laying out these ranges in the proper order and in a seamless
> presentation. The editing of such data would also be straightforward. The
> bit view has a bit editor associated with it, allowing inline modification
> of bits, just like you would expect in the hex editor.
>
> Where it gets fuzzy is editing in a representation different from the
> display, eg. editing bits from a byte view.
>
> > a way to expand a single byte into a small presentation
> > of 8 bits, allowing editing of just those 8 bits individually
>
> I can visualize this a few ways, not sure of what is the best for
> usability. A properties pane type component when a byte is selected might
> be the right one to start with.
>
> > Perhaps that is what you meant by Mask/Set operations?
>
> I was only considering a byte view; "Mask" was referring to bit
> manipulation of each byte in a range, "Set" as a verb, simply to update the
> value of an existing byte.
>
> Your thoughts about the bit based views fill in the gaps around bits and we
> can consider similar context operations in that view as those I mentioned
> for the byte view.
>
> > I would add to the "parse until this byte", or "start from this byte" to
> > enable narrowing at both the start and end - parse just these bytes
>
> Yeah I like this. The interface between hex editor and debugger commands
> is going to be an important extension point.
>
>
>
>
> On Mon, Nov 1, 2021 at 10:31 PM Mike Beckerle <mb...@apache.org>
> wrote:
>
> > This is a good list.
> >
> > One thing I think is important is that often one is dealing with hex
> data,
> > but one needs to consider bit fields that do not respect byte boundaries.
> >
> > I suggest we need a way to expand a single byte into a small presentation
> > of 8 bits, allowing editing of just those 8 bits individually, is of
> value
> > as part of a hex-editor. Perhaps that is what you meant by Mask/Set
> > operations? I think switching to a full-fledged all 1's and 0's display
> > mode is only for entirely non-byte-oriented data. Anything byte-oriented
> > people users will want to use hex, and occasionally if they need to flip
> > bits, a way for them to expand a byte or small run of bytes to 1's and
> 0's,
> > but then collapse back to hex is likely very helpful.
> >
> > But I also suggest creating a minimal hex editor version first, then we
> get
> > experience with it.
> >
> > Any little paper scribble exercises we find ourselves having to do on the
> > side of using the editor, those are good candidates for things the UI
> > should directly support.
> >
> > E.g. I have written things down on paper or in a text editor like:
> >
> > BF.32.A5.AC.(01|10 1101.101|1 0001)
> >
> > In a real UI these distinctions could be done quite differently.
> > But the distinctions I'm making are dots separating bytes, pipes
> separating
> > bit fields of length 2, 9, and 5 respectively, and parentheses indicating
> > bytes expanded out to a bits region for display, e.g., so that 01 for 2
> > bits isn't confused with hex 01 8 bits, and here I have each nibble of 4
> > bits space-separated.
> >
> > Ultimately, we need a graphical means providing:
> > (a) a way of escaping from hex to bits for just a small region of the
> data
> > where you care about the partial byte fields.
> > (b) a way of setting off bit fields from each other, that doesn't
> entirely
> > lose the separation of hex digits.
> >
> > Then if say, that right-most bit field value is incorrect, you should be
> > able to set editing focus on a bit and flip it.
> >
> > I would add to the "parse until this byte", or "start from this byte" to
> > enable narrowing at both the start and end - parse just these bytes (in
> an
> > identified region by start and end). The use case I have in mind for
> this
> > is sort of like unit testing. You narrow the data to just one part, then
> > you specify to parse it not with the root element of the DFDL schema, but
> > with a sub-element that you want to test.
> >
> > On Mon, Nov 1, 2021 at 6:19 AM John Wass <jw...@gmail.com> wrote:
> >
> > > Some thoughts on ways the hex editor would interact with an input
> stream
> > > and the debugger. I visualize this using a context sensitive menu in
> the
> > > hex editor as an entrypoint, where these are some of the operations
> that
> > > could be offered.
> > >
> > > 1. Add/Delete/Mask/Set
> > > * Individual bytes
> > > * Blocks of bytes
> > > 2. Copy/Paste bytes
> > > * Use system clipboard for interoperability
> > > 3. Find/Replace bytes
> > > * In a selection
> > > * Across entire file
> > > 4. Set breakpoints on bytes
> > > * Stops execution at related point in schema
> > > * Would require custom additions to DAP backend
> > > 5. Set run options on bytes
> > > * Debug-to this byte (and break)
> > > * Start from this byte (skipping previous)
> > >
> > > The idea with breakpoints and run operations is that they would behave
> > as a
> > > normal breakpoint in code would. I don't know if this is functionality
> > for
> > > the first pass of the editor, but it is definitely something to keep in
> > > mind while designing that first pass.
> > >
> > > What other operations could be supported?
> > >
> >
>
Re: Hex editor operations
Posted by John Wass <jw...@gmail.com>.
Yep, I agree with all the bits about bits.
The display of non-byte delimited data is covered by the concept of a
composable UI that allows for rendering ranges within the file differently
while laying out these ranges in the proper order and in a seamless
presentation. The editing of such data would also be straightforward. The
bit view has a bit editor associated with it, allowing inline modification
of bits, just like you would expect in the hex editor.
Where it gets fuzzy is editing in a representation different from the
display, eg. editing bits from a byte view.
> a way to expand a single byte into a small presentation
> of 8 bits, allowing editing of just those 8 bits individually
I can visualize this a few ways, not sure of what is the best for
usability. A properties pane type component when a byte is selected might
be the right one to start with.
> Perhaps that is what you meant by Mask/Set operations?
I was only considering a byte view; "Mask" was referring to bit
manipulation of each byte in a range, "Set" as a verb, simply to update the
value of an existing byte.
Your thoughts about the bit based views fill in the gaps around bits and we
can consider similar context operations in that view as those I mentioned
for the byte view.
> I would add to the "parse until this byte", or "start from this byte" to
> enable narrowing at both the start and end - parse just these bytes
Yeah I like this. The interface between hex editor and debugger commands
is going to be an important extension point.
On Mon, Nov 1, 2021 at 10:31 PM Mike Beckerle <mb...@apache.org> wrote:
> This is a good list.
>
> One thing I think is important is that often one is dealing with hex data,
> but one needs to consider bit fields that do not respect byte boundaries.
>
> I suggest we need a way to expand a single byte into a small presentation
> of 8 bits, allowing editing of just those 8 bits individually, is of value
> as part of a hex-editor. Perhaps that is what you meant by Mask/Set
> operations? I think switching to a full-fledged all 1's and 0's display
> mode is only for entirely non-byte-oriented data. Anything byte-oriented
> people users will want to use hex, and occasionally if they need to flip
> bits, a way for them to expand a byte or small run of bytes to 1's and 0's,
> but then collapse back to hex is likely very helpful.
>
> But I also suggest creating a minimal hex editor version first, then we get
> experience with it.
>
> Any little paper scribble exercises we find ourselves having to do on the
> side of using the editor, those are good candidates for things the UI
> should directly support.
>
> E.g. I have written things down on paper or in a text editor like:
>
> BF.32.A5.AC.(01|10 1101.101|1 0001)
>
> In a real UI these distinctions could be done quite differently.
> But the distinctions I'm making are dots separating bytes, pipes separating
> bit fields of length 2, 9, and 5 respectively, and parentheses indicating
> bytes expanded out to a bits region for display, e.g., so that 01 for 2
> bits isn't confused with hex 01 8 bits, and here I have each nibble of 4
> bits space-separated.
>
> Ultimately, we need a graphical means providing:
> (a) a way of escaping from hex to bits for just a small region of the data
> where you care about the partial byte fields.
> (b) a way of setting off bit fields from each other, that doesn't entirely
> lose the separation of hex digits.
>
> Then if say, that right-most bit field value is incorrect, you should be
> able to set editing focus on a bit and flip it.
>
> I would add to the "parse until this byte", or "start from this byte" to
> enable narrowing at both the start and end - parse just these bytes (in an
> identified region by start and end). The use case I have in mind for this
> is sort of like unit testing. You narrow the data to just one part, then
> you specify to parse it not with the root element of the DFDL schema, but
> with a sub-element that you want to test.
>
> On Mon, Nov 1, 2021 at 6:19 AM John Wass <jw...@gmail.com> wrote:
>
> > Some thoughts on ways the hex editor would interact with an input stream
> > and the debugger. I visualize this using a context sensitive menu in the
> > hex editor as an entrypoint, where these are some of the operations that
> > could be offered.
> >
> > 1. Add/Delete/Mask/Set
> > * Individual bytes
> > * Blocks of bytes
> > 2. Copy/Paste bytes
> > * Use system clipboard for interoperability
> > 3. Find/Replace bytes
> > * In a selection
> > * Across entire file
> > 4. Set breakpoints on bytes
> > * Stops execution at related point in schema
> > * Would require custom additions to DAP backend
> > 5. Set run options on bytes
> > * Debug-to this byte (and break)
> > * Start from this byte (skipping previous)
> >
> > The idea with breakpoints and run operations is that they would behave
> as a
> > normal breakpoint in code would. I don't know if this is functionality
> for
> > the first pass of the editor, but it is definitely something to keep in
> > mind while designing that first pass.
> >
> > What other operations could be supported?
> >
>
Re: Hex editor operations
Posted by Mike Beckerle <mb...@apache.org>.
This is a good list.
One thing I think is important is that often one is dealing with hex data,
but one needs to consider bit fields that do not respect byte boundaries.
I suggest we need a way to expand a single byte into a small presentation
of 8 bits, allowing editing of just those 8 bits individually, is of value
as part of a hex-editor. Perhaps that is what you meant by Mask/Set
operations? I think switching to a full-fledged all 1's and 0's display
mode is only for entirely non-byte-oriented data. Anything byte-oriented
people users will want to use hex, and occasionally if they need to flip
bits, a way for them to expand a byte or small run of bytes to 1's and 0's,
but then collapse back to hex is likely very helpful.
But I also suggest creating a minimal hex editor version first, then we get
experience with it.
Any little paper scribble exercises we find ourselves having to do on the
side of using the editor, those are good candidates for things the UI
should directly support.
E.g. I have written things down on paper or in a text editor like:
BF.32.A5.AC.(01|10 1101.101|1 0001)
In a real UI these distinctions could be done quite differently.
But the distinctions I'm making are dots separating bytes, pipes separating
bit fields of length 2, 9, and 5 respectively, and parentheses indicating
bytes expanded out to a bits region for display, e.g., so that 01 for 2
bits isn't confused with hex 01 8 bits, and here I have each nibble of 4
bits space-separated.
Ultimately, we need a graphical means providing:
(a) a way of escaping from hex to bits for just a small region of the data
where you care about the partial byte fields.
(b) a way of setting off bit fields from each other, that doesn't entirely
lose the separation of hex digits.
Then if say, that right-most bit field value is incorrect, you should be
able to set editing focus on a bit and flip it.
I would add to the "parse until this byte", or "start from this byte" to
enable narrowing at both the start and end - parse just these bytes (in an
identified region by start and end). The use case I have in mind for this
is sort of like unit testing. You narrow the data to just one part, then
you specify to parse it not with the root element of the DFDL schema, but
with a sub-element that you want to test.
On Mon, Nov 1, 2021 at 6:19 AM John Wass <jw...@gmail.com> wrote:
> Some thoughts on ways the hex editor would interact with an input stream
> and the debugger. I visualize this using a context sensitive menu in the
> hex editor as an entrypoint, where these are some of the operations that
> could be offered.
>
> 1. Add/Delete/Mask/Set
> * Individual bytes
> * Blocks of bytes
> 2. Copy/Paste bytes
> * Use system clipboard for interoperability
> 3. Find/Replace bytes
> * In a selection
> * Across entire file
> 4. Set breakpoints on bytes
> * Stops execution at related point in schema
> * Would require custom additions to DAP backend
> 5. Set run options on bytes
> * Debug-to this byte (and break)
> * Start from this byte (skipping previous)
>
> The idea with breakpoints and run operations is that they would behave as a
> normal breakpoint in code would. I don't know if this is functionality for
> the first pass of the editor, but it is definitely something to keep in
> mind while designing that first pass.
>
> What other operations could be supported?
>