You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2020/07/17 21:21:21 UTC
[Format] Feature field in Schema
Hello,
A bit too late, I noticed the new "features" field is defined on the
Schema table, while the "version" field is defined on the Message table.
Since both fields have closely related purposes (notify the reader of
the conventions used in the stream), I'm a bit surprised that they're
present at different places. Is there a reason I missed for this
seeming inconsistency?
Regards
Antoine.
Re: [Format] Feature field in Schema
Posted by Antoine Pitrou <an...@python.org>.
Yes, that's ok with me.
Regards
Antoine.
Le 18/07/2020 à 18:30, Micah Kornfield a écrit :
> Antoine are you OK with leaving as is?
>
> On Fri, Jul 17, 2020 at 3:06 PM Wes McKinney <we...@gmail.com> wrote:
>
>> My position is that:
>>
>> * Features only needs to be set with the Schema message, it wouldn't
>> be necessary or useful to set it for other message types
>> * The metadata version may serve a purpose beyond indicating features
>> (and it has in the past already)
>> * Thus, it isn't necessarily inconsistent to have these fields at
>> different "levels" of the hierarchy, and having "features" be null in
>> non-Schema message types might be a different (better or worse?)
>> inconsistency
>>
>> So I would be -0 on changing anything, but maybe there are other
>> concerns that I haven't listed
>>
>> On Fri, Jul 17, 2020 at 4:47 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>>
>>> I think this was overlooked. Schema made more sense to me because I was
>>> intending it to be at most once per stream. If we can come to agreement
>> I
>>> can open a PR to change it. But we would need a new release candidate
>>> (this can't wait until next release)
>>>
>>> On Friday, July 17, 2020, Antoine Pitrou <an...@python.org> wrote:
>>>
>>>>
>>>> Hello,
>>>>
>>>> A bit too late, I noticed the new "features" field is defined on the
>>>> Schema table, while the "version" field is defined on the Message
>> table.
>>>> Since both fields have closely related purposes (notify the reader of
>>>> the conventions used in the stream), I'm a bit surprised that they're
>>>> present at different places. Is there a reason I missed for this
>>>> seeming inconsistency?
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>
>
Re: [Format] Feature field in Schema
Posted by Micah Kornfield <em...@gmail.com>.
Antoine are you OK with leaving as is?
On Fri, Jul 17, 2020 at 3:06 PM Wes McKinney <we...@gmail.com> wrote:
> My position is that:
>
> * Features only needs to be set with the Schema message, it wouldn't
> be necessary or useful to set it for other message types
> * The metadata version may serve a purpose beyond indicating features
> (and it has in the past already)
> * Thus, it isn't necessarily inconsistent to have these fields at
> different "levels" of the hierarchy, and having "features" be null in
> non-Schema message types might be a different (better or worse?)
> inconsistency
>
> So I would be -0 on changing anything, but maybe there are other
> concerns that I haven't listed
>
> On Fri, Jul 17, 2020 at 4:47 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > I think this was overlooked. Schema made more sense to me because I was
> > intending it to be at most once per stream. If we can come to agreement
> I
> > can open a PR to change it. But we would need a new release candidate
> > (this can't wait until next release)
> >
> > On Friday, July 17, 2020, Antoine Pitrou <an...@python.org> wrote:
> >
> > >
> > > Hello,
> > >
> > > A bit too late, I noticed the new "features" field is defined on the
> > > Schema table, while the "version" field is defined on the Message
> table.
> > > Since both fields have closely related purposes (notify the reader of
> > > the conventions used in the stream), I'm a bit surprised that they're
> > > present at different places. Is there a reason I missed for this
> > > seeming inconsistency?
> > >
> > > Regards
> > >
> > > Antoine.
> > >
>
Re: [Format] Feature field in Schema
Posted by Wes McKinney <we...@gmail.com>.
My position is that:
* Features only needs to be set with the Schema message, it wouldn't
be necessary or useful to set it for other message types
* The metadata version may serve a purpose beyond indicating features
(and it has in the past already)
* Thus, it isn't necessarily inconsistent to have these fields at
different "levels" of the hierarchy, and having "features" be null in
non-Schema message types might be a different (better or worse?)
inconsistency
So I would be -0 on changing anything, but maybe there are other
concerns that I haven't listed
On Fri, Jul 17, 2020 at 4:47 PM Micah Kornfield <em...@gmail.com> wrote:
>
> I think this was overlooked. Schema made more sense to me because I was
> intending it to be at most once per stream. If we can come to agreement I
> can open a PR to change it. But we would need a new release candidate
> (this can't wait until next release)
>
> On Friday, July 17, 2020, Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > A bit too late, I noticed the new "features" field is defined on the
> > Schema table, while the "version" field is defined on the Message table.
> > Since both fields have closely related purposes (notify the reader of
> > the conventions used in the stream), I'm a bit surprised that they're
> > present at different places. Is there a reason I missed for this
> > seeming inconsistency?
> >
> > Regards
> >
> > Antoine.
> >
Re: [Format] Feature field in Schema
Posted by Micah Kornfield <em...@gmail.com>.
I think this was overlooked. Schema made more sense to me because I was
intending it to be at most once per stream. If we can come to agreement I
can open a PR to change it. But we would need a new release candidate
(this can't wait until next release)
On Friday, July 17, 2020, Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> A bit too late, I noticed the new "features" field is defined on the
> Schema table, while the "version" field is defined on the Message table.
> Since both fields have closely related purposes (notify the reader of
> the conventions used in the stream), I'm a bit surprised that they're
> present at different places. Is there a reason I missed for this
> seeming inconsistency?
>
> Regards
>
> Antoine.
>
Re: [Format] Feature field in Schema
Posted by Wes McKinney <we...@gmail.com>.
Any of the dependent message types constituting an IPC stream or file
are not interpretable without the Schema message, so the Schema
basically "governs" the other messages. So it is sufficient to examine
the features only once at schema resolution time, and attaching
features to a RecordBatch message is not necessary (and maybe it would
be weird to have the field set only in Schema message types but then
null for other message types). That's perhaps not a completely
persuasive argument about why version and features are not contained
in the same Flatbuffers table.
Another argument I can see is that the version number at the Message
level provides an "escape hatch" for non-forward-compatible
alterations of message types, even at the binary level. Having it at
the Message level also spares applications from having to set the
version in multiple places rather than a single place.
On Fri, Jul 17, 2020 at 4:23 PM Antoine Pitrou <an...@python.org> wrote:
>
>
> Hello,
>
> A bit too late, I noticed the new "features" field is defined on the
> Schema table, while the "version" field is defined on the Message table.
> Since both fields have closely related purposes (notify the reader of
> the conventions used in the stream), I'm a bit surprised that they're
> present at different places. Is there a reason I missed for this
> seeming inconsistency?
>
> Regards
>
> Antoine.