You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dave Latham <la...@davelink.net> on 2015/04/14 00:16:27 UTC
toStringBinary output is painful
Wish I had started this conversation 5 years ago...
When we're using binary data, especially in row keys (and therefore
region boundaries) the output created by toStringBinary is very
painful to use:
- mix of ascii / hex representation is trouble
- it's quite long (4 characters per binary byte)
- it mixes in punctuation characters (quotes are not good when
pasting in to shell commands)
Does this bother anyone else? Is it too late to do anything about it
due to compatibility fears?
Can we change to something that is always binary e.g. hex?
0x746f537472696e6742696e617279206d616b657320636c75737465727320637279
Or if we have to auto-detect, only use ascii if the entire string is ascii?
Dave
Re: toStringBinary output is painful
Posted by hongbin ma <ma...@apache.org>.
+1
I found the mix of ascii/hex very painful when I need to compare two binary
keys/values
On Tue, Apr 14, 2015 at 6:16 AM, Dave Latham <la...@davelink.net> wrote:
> Wish I had started this conversation 5 years ago...
>
> When we're using binary data, especially in row keys (and therefore
> region boundaries) the output created by toStringBinary is very
> painful to use:
> - mix of ascii / hex representation is trouble
> - it's quite long (4 characters per binary byte)
> - it mixes in punctuation characters (quotes are not good when
> pasting in to shell commands)
>
> Does this bother anyone else? Is it too late to do anything about it
> due to compatibility fears?
>
> Can we change to something that is always binary e.g. hex?
> 0x746f537472696e6742696e617279206d616b657320636c75737465727320637279
>
> Or if we have to auto-detect, only use ascii if the entire string is ascii?
>
> Dave
>
--
Regards,
*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone
Re: toStringBinary output is painful
Posted by Sean Busbey <bu...@cloudera.com>.
It does not bother just you. :)
Can we bound where all this shows up? In most cases we can address
compatibility concerns by making a new API and transitioning to it as the
guidelines allow for breakage.
That generally means sooner is better than later; our next big effort to
make sure obsolete things are deprecated will be whenever the push for 2.0
happens. So far that looks like somewhere in Aug-Dec.
--
Sean
On Apr 13, 2015 5:18 PM, "Dave Latham" <la...@davelink.net> wrote:
> Wish I had started this conversation 5 years ago...
>
> When we're using binary data, especially in row keys (and therefore
> region boundaries) the output created by toStringBinary is very
> painful to use:
> - mix of ascii / hex representation is trouble
> - it's quite long (4 characters per binary byte)
> - it mixes in punctuation characters (quotes are not good when
> pasting in to shell commands)
>
> Does this bother anyone else? Is it too late to do anything about it
> due to compatibility fears?
>
> Can we change to something that is always binary e.g. hex?
> 0x746f537472696e6742696e617279206d616b657320636c75737465727320637279
>
> Or if we have to auto-detect, only use ascii if the entire string is ascii?
>
> Dave
>
Re: toStringBinary output is painful
Posted by Dave Latham <la...@davelink.net>.
> I believe toStringBinary does all ascii if input ascii-only.
Right, but it will also mix ascii range characters with binary.
> Our output was in part shaped by ruby binary String representation. It was
> thought useful that you could copy from shell and find in UI, and
> vice-versa. I don't think many make use of this fact.
Interesting - didn't realize the motivation.
> We could change toStringBinary format. While rarely needed (now), probably
> best if the format allows us get back to the original bytes.
Absolutely - this would still be a requirement for us. It should be a
reversible encoding.
> We could also work on hiding the uglyness from users since it hurts relying
> on region id instead and/or making smarter key choices... doing minimal
> necessary rather than full row key (we have the code already in our hfile
> index).
That would help with some cases, especially shell admin commands, but
I don't think is enough.
> Where is the pain in particular Dave?
>
> Using the shell or looking at UIs and logs?
It's a dull ache in many places which is why we have tolerated it for
years. Region names and boundaries (especially ones auto generated
from splits) make the UI annoying to read and error prone when
copy/pasting the data around to other places that aren't as happy with
the whitespace or punctuation. Region server logs. Verify
replication logs.
> What if we hid the uglyness.
> toStringBinary will do ascii if all ascii
That was one option I suggested. There may still be unusual cases of
binary keys that happen to be ascii, but they would be much rarer.
For us internally, I'd just do hex everywhere, but I understand if
enough people use ascii data encodings and find it helpful for those
to show up in the UI/logs.
Dave
Re: toStringBinary output is painful
Posted by Stack <st...@duboce.net>.
On Mon, Apr 13, 2015 at 3:16 PM, Dave Latham <la...@davelink.net> wrote:
> Wish I had started this conversation 5 years ago...
>
> When we're using binary data, especially in row keys (and therefore
> region boundaries) the output created by toStringBinary is very
> painful to use:
> - mix of ascii / hex representation is trouble
> - it's quite long (4 characters per binary byte)
> - it mixes in punctuation characters (quotes are not good when
> pasting in to shell commands)
>
> Does this bother anyone else? Is it too late to do anything about it
> due to compatibility fears?
> Can we change to something that is always binary e.g. hex?
> 0x746f537472696e6742696e617279206d616b657320636c75737465727320637279
>
> Or if we have to auto-detect, only use ascii if the entire string is ascii?
>
I believe toStringBinary does all ascii if input ascii-only.
Our output was in part shaped by ruby binary String representation. It was
thought useful that you could copy from shell and find in UI, and
vice-versa. I don't think many make use of this fact.
We could change toStringBinary format. While rarely needed (now), probably
best if the format allows us get back to the original bytes.
We could also work on hiding the uglyness from users since it hurts relying
on region id instead and/or making smarter key choices... doing minimal
necessary rather than full row key (we have the code already in our hfile
index).
St.Ack
Where is the pain in particular Dave?
Using the shell or looking at UIs and logs?
What if we hid the uglyness.
toStringBinary will do ascii if all ascii