You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2019/11/07 19:24:50 UTC
[DISCUSS] C data interface updated
Hello,
The C data interface spec was updated following a suggestion
by Wes and Uwe:
https://github.com/apache/arrow/blob/3173f88dfa32ce3296a121b032f351e089888601/docs/source/format/CDataInterface.rst
The metadata encoding was changed. It does not use JSON anymore
but a very simple binary encoding. Quoting the spec:
This string is not null-terminated but follows a specific format::
int32: number of key/value pairs (noted N below)
int32: byte length of key 0
key 0 (not null-terminated)
int32: byte length of value 0
value 0 (not null-terminated)
...
int32: byte length of key N - 1
key N - 1 (not null-terminated)
int32: byte length of value N - 1
value N - 1 (not null-terminated)
For example, the metadata ``[('key1', 'value1')]`` is encoded as::
\x01\x00\x00\x00\x04\x00\x00\x00key1\x06\x00\x00\x00value1
The C++ draft implementation now supports metadata exporting and importing:
https://github.com/apache/arrow/pull/5608
Regards
Antoine.
Re: [DISCUSS] C data interface updated
Posted by Wes McKinney <we...@gmail.com>.
I started a vote and left comments (mostly clarifications) on the
specification PR. I'm reviewing the C++ patch and will post comments
when I can
On Wed, Nov 13, 2019 at 4:39 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> Yes, we should probably need to hold a vote at some point. Perhaps wait
> a week or so for further comments.
>
> Regards
>
> Antoine.
>
>
> Le 08/11/2019 à 03:00, Wes McKinney a écrit :
> > Thanks Antoine.
> >
> > Do we need to hold a vote to adopt the C API? I will see if I can
> > review the C++ implementation prior to leaving for ~1 week of vacation
> > next Tuesday
> >
> > On Thu, Nov 7, 2019 at 1:24 PM Antoine Pitrou <an...@python.org> wrote:
> >>
> >>
> >> Hello,
> >>
> >> The C data interface spec was updated following a suggestion
> >> by Wes and Uwe:
> >> https://github.com/apache/arrow/blob/3173f88dfa32ce3296a121b032f351e089888601/docs/source/format/CDataInterface.rst
> >>
> >> The metadata encoding was changed. It does not use JSON anymore
> >> but a very simple binary encoding. Quoting the spec:
> >>
> >> This string is not null-terminated but follows a specific format::
> >>
> >> int32: number of key/value pairs (noted N below)
> >> int32: byte length of key 0
> >> key 0 (not null-terminated)
> >> int32: byte length of value 0
> >> value 0 (not null-terminated)
> >> ...
> >> int32: byte length of key N - 1
> >> key N - 1 (not null-terminated)
> >> int32: byte length of value N - 1
> >> value N - 1 (not null-terminated)
> >>
> >> For example, the metadata ``[('key1', 'value1')]`` is encoded as::
> >>
> >> \x01\x00\x00\x00\x04\x00\x00\x00key1\x06\x00\x00\x00value1
> >>
> >>
> >> The C++ draft implementation now supports metadata exporting and importing:
> >> https://github.com/apache/arrow/pull/5608
> >>
> >> Regards
> >>
> >> Antoine.
Re: [DISCUSS] C data interface updated
Posted by Antoine Pitrou <an...@python.org>.
Yes, we should probably need to hold a vote at some point. Perhaps wait
a week or so for further comments.
Regards
Antoine.
Le 08/11/2019 à 03:00, Wes McKinney a écrit :
> Thanks Antoine.
>
> Do we need to hold a vote to adopt the C API? I will see if I can
> review the C++ implementation prior to leaving for ~1 week of vacation
> next Tuesday
>
> On Thu, Nov 7, 2019 at 1:24 PM Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Hello,
>>
>> The C data interface spec was updated following a suggestion
>> by Wes and Uwe:
>> https://github.com/apache/arrow/blob/3173f88dfa32ce3296a121b032f351e089888601/docs/source/format/CDataInterface.rst
>>
>> The metadata encoding was changed. It does not use JSON anymore
>> but a very simple binary encoding. Quoting the spec:
>>
>> This string is not null-terminated but follows a specific format::
>>
>> int32: number of key/value pairs (noted N below)
>> int32: byte length of key 0
>> key 0 (not null-terminated)
>> int32: byte length of value 0
>> value 0 (not null-terminated)
>> ...
>> int32: byte length of key N - 1
>> key N - 1 (not null-terminated)
>> int32: byte length of value N - 1
>> value N - 1 (not null-terminated)
>>
>> For example, the metadata ``[('key1', 'value1')]`` is encoded as::
>>
>> \x01\x00\x00\x00\x04\x00\x00\x00key1\x06\x00\x00\x00value1
>>
>>
>> The C++ draft implementation now supports metadata exporting and importing:
>> https://github.com/apache/arrow/pull/5608
>>
>> Regards
>>
>> Antoine.
Re: [DISCUSS] C data interface updated
Posted by Wes McKinney <we...@gmail.com>.
Thanks Antoine.
Do we need to hold a vote to adopt the C API? I will see if I can
review the C++ implementation prior to leaving for ~1 week of vacation
next Tuesday
On Thu, Nov 7, 2019 at 1:24 PM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hello,
>
> The C data interface spec was updated following a suggestion
> by Wes and Uwe:
> https://github.com/apache/arrow/blob/3173f88dfa32ce3296a121b032f351e089888601/docs/source/format/CDataInterface.rst
>
> The metadata encoding was changed. It does not use JSON anymore
> but a very simple binary encoding. Quoting the spec:
>
> This string is not null-terminated but follows a specific format::
>
> int32: number of key/value pairs (noted N below)
> int32: byte length of key 0
> key 0 (not null-terminated)
> int32: byte length of value 0
> value 0 (not null-terminated)
> ...
> int32: byte length of key N - 1
> key N - 1 (not null-terminated)
> int32: byte length of value N - 1
> value N - 1 (not null-terminated)
>
> For example, the metadata ``[('key1', 'value1')]`` is encoded as::
>
> \x01\x00\x00\x00\x04\x00\x00\x00key1\x06\x00\x00\x00value1
>
>
> The C++ draft implementation now supports metadata exporting and importing:
> https://github.com/apache/arrow/pull/5608
>
> Regards
>
> Antoine.