You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Pavel Martynov <mr...@gmail.com> on 2019/06/11 13:08:25 UTC
Kudu CLI tool JSON format
Hi, guys!
We trying to use an output of "kudu cluster ksck master -ksck_format
json_compact" for integration with our monitoring system and hit a little
strange. Some part of output can't be read as UTF-8 with Python 3:
$ kudu cluster ksck master -ksck_format json_compact > kudu.json
$ python
with open(' kudu.json', mode='rb') as file:
bs = file.read()
bs.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position
705196: invalid start byte
There how SublimeText shows this block of text:
https://yadi.sk/i/4zpWKZ37iP8OEA
As you can see kudu tool encodes zeros as \u0000, but don't encode some
other non-text bytes.
What do you think about it?
--
with best regards, Pavel Martynov
Re: Kudu CLI tool JSON format
Posted by Adar Lieber-Dembo <ad...@cloudera.com>.
Thanks for the report. I filed KUDU-2845 to track the issue.
On Tue, Jun 11, 2019 at 9:44 AM Todd Lipcon <to...@cloudera.com> wrote:
>
> I guess the issue is that we use rapidjson's 'String' support to write out C++ strings, which are binary data, not valid UTF8. That's somewhat incorrect of us, and we should be base64-encoding such binary data.
>
> Fixing this is a little bit incompatible, but for something like partition keys I think we probably should do it anyway and release note it, considering partition keys are quite likely to be invalid UTF8.
>
> -Todd
>
> On Tue, Jun 11, 2019 at 6:08 AM Pavel Martynov <mr...@gmail.com> wrote:
>>
>> Hi, guys!
>>
>> We trying to use an output of "kudu cluster ksck master -ksck_format json_compact" for integration with our monitoring system and hit a little strange. Some part of output can't be read as UTF-8 with Python 3:
>> $ kudu cluster ksck master -ksck_format json_compact > kudu.json
>> $ python
>> with open(' kudu.json', mode='rb') as file:
>> bs = file.read()
>> bs.decode('utf-8')
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 705196: invalid start byte
>>
>> There how SublimeText shows this block of text: https://yadi.sk/i/4zpWKZ37iP8OEA
>> As you can see kudu tool encodes zeros as \u0000, but don't encode some other non-text bytes.
>>
>> What do you think about it?
>>
>> --
>> with best regards, Pavel Martynov
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Kudu CLI tool JSON format
Posted by Todd Lipcon <to...@cloudera.com>.
I guess the issue is that we use rapidjson's 'String' support to write out
C++ strings, which are binary data, not valid UTF8. That's somewhat
incorrect of us, and we should be base64-encoding such binary data.
Fixing this is a little bit incompatible, but for something like partition
keys I think we probably should do it anyway and release note it,
considering partition keys are quite likely to be invalid UTF8.
-Todd
On Tue, Jun 11, 2019 at 6:08 AM Pavel Martynov <mr...@gmail.com> wrote:
> Hi, guys!
>
> We trying to use an output of "kudu cluster ksck master -ksck_format
> json_compact" for integration with our monitoring system and hit a little
> strange. Some part of output can't be read as UTF-8 with Python 3:
> $ kudu cluster ksck master -ksck_format json_compact > kudu.json
> $ python
> with open(' kudu.json', mode='rb') as file:
> bs = file.read()
> bs.decode('utf-8')
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position
> 705196: invalid start byte
>
> There how SublimeText shows this block of text:
> https://yadi.sk/i/4zpWKZ37iP8OEA
> As you can see kudu tool encodes zeros as \u0000, but don't encode some
> other non-text bytes.
>
> What do you think about it?
>
> --
> with best regards, Pavel Martynov
>
--
Todd Lipcon
Software Engineer, Cloudera