[INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi guys, hope you are well!

Judging from the Flight API
<https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461>
and
from the documentation/examples out there, it seems like data schema is
supposed to be fixed per stream in ArrowFlight(which is also aligned with
corresponding IPC stream writers/readers).
Wondering if the community has evaluated the necessity/possibility of
supporting schema changes within a single stream(I do recall seeing a
discussion on this somewhere but can't find it)?

Cheers,
Gosh

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Phillip Cloud <cp...@gmail.com>]

I don't think memcpy is feasible. The bytes may be different for different
languages.

Many languages' compilers reorder struct fields and pad structs for
efficiency reasons so the bytes in metadata coming from language X may be
meaningless to language Y.


On Wed, Feb 9, 2022, 08:41 Dewey Dunnington <de...@voltrondata.com> wrote:

> I'm new to this discussion and Arrow generally and appreciate Joris
> bringing this up. While the forward- and backward-compatability of this is
> over my head, I think that it is important to provide a path for extension
> types to have serialized metadata as binary because the alternatives are
> (in my opinion) not great and might lead to extension developers inventing
> their own key/value encoding.
>
> With binary on the table there are options like protobuf, flatbuffers, or
> memcpy of a struct that extension type implementations can use depending on
> the complexity of the extension metadata that needs to be communicated and
> whether or not the metadata is intended to be used in compiled code (i.e.,
> are those values needed to do anything useful with the bytes in the
> buffers?).
>
> Without binary on the table, an extension type implementation has to vendor
> in a JSON reader/writer, vendor in a base64 encoder/decoder, or invent some
> custom serialization that's easier to parse than either of those.
>
> Cheers,
>
> -dewey
>
> On Fri, Feb 4, 2022 at 7:06 AM Joris Van den Bossche <
> jorisvandenbossche@gmail.com> wrote:
>
> > Reviving this thread, I don't think anything happened in the meantime on
> > this topic?
> >
> > From rereading the thread, it seems David mentioned two possible ideas:
> > - A new [byte] binary_value field in the existing KeyValue type, next to
> > the existing string value field. And if you have valid string metadata,
> the
> > same value can be put in both fields by reusing the offset
> > - Relax the existing string value field to [byte], but also require
> > implementations to null-terminate binary values regardless to ensure
> > compatibility for old readers
> >  - And in addition to those options, there is also the safest option of
> > adding a new BinaryKeyValue type and add that next to KeyValue everywhere
> > this is used.
> >
> > And we need to further assess which option is viable given the
> > forward/backward compatibility constraints?
> >
> > I want to point out though that in pyarrow we already _do_ use binary
> > metadata for extension types for several years. The built-in
> > PyExtensionType class uses a pickle dump as the serialized metadata (so
> > which is put in the ARROW:extension:metadata key in the key-value
> metadata
> > of the Field), and the bytes from a pickle dump are certainly not valid
> > UTF-8 I think.
> > I don't know how much this is used in the wild. In pandas, our arrow
> > extension types don't use this pickle-based PyExtensionType but define
> > their own serialization which currently returns a valid UTF-8 string. But
> > for example the Ray project seems to use PyExtensionType (
> >
> >
> https://docs.ray.io/en/latest/_modules/ray/data/extensions/tensor_extension.html#ArrowTensorType
> > ).
> >
> > Joris
> >
> > On Thu, 2 Sept 2021 at 07:01, Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > It still sounds like adding a new type might be the safest approach
> (and
> > > marking the old type as discouraged).
> > >
> > > On Mon, Aug 23, 2021 at 11:18 AM David Li <li...@apache.org> wrote:
> > >
> > > > I believe so.
> > > >
> > > > The encoding of a string in Flatbuffers is [byte] with a null
> > terminator
> > > > not included in the length, so old files should still be readable
> (they
> > > > would simply not see the terminator anymore). And conversely,
> > continuing
> > > to
> > > > write the null terminator means new files should still be readable by
> > > > existing implementations, so long as they don't have binary metadata
> > (but
> > > > we could increment the metadata version before allowing that). There
> > > still
> > > > remains the issue of non-UTF8 data inside the "string" value; on that
> > > > point, this is a non-answer (or I guess, it canonicalizes the current
> > > > unintentional behavior).
> > > >
> > > > It would complicate the Java API as that currently works with
> Strings,
> > > but
> > > > that will be an issue for any attempt to add binary metadata.
> > > >
> > > > -David
> > > >
> > > > On Mon, Aug 23, 2021, at 13:40, Antoine Pitrou wrote:
> > > > >
> > > > > Le 23/08/2021 à 17:52, David Li a écrit :
> > > > > > Another way forward might be to relax the value type to [byte],
> but
> > > > also require implementations to null-terminate binary values
> > regardless.
> > > > The C++ Flatbuffers implementation does this already [1] (though not
> > the
> > > > Java one [2]). Old implementations validating UTF8-ness would still
> be
> > > > unable to read anything with binary metadata (which is the case
> they're
> > > > already in), but implementations just validating for null-termination
> > > would
> > > > continue to work. And again, I believe this is in-spec for
> Flatbuffers
> > > > since string lengths don't include the null terminator.
> > > > > >
> > > > > > [1]:
> > > >
> > >
> >
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670
> > > > > > [2]:
> > > >
> > >
> >
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637
> > > > > > Though it seems we could emulate C++ here by calling addByte(0)
> > > > ourselves before createByteVector.
> > > > >
> > > > > Would that allow for both forwards and backwards compatibility when
> > the
> > > > > metadata is valid UTF8?
> > > > >
> > > > > Regards
> > > > >
> > > > > Antoine.
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Dewey Dunnington <de...@voltrondata.com>]

I'm new to this discussion and Arrow generally and appreciate Joris
bringing this up. While the forward- and backward-compatability of this is
over my head, I think that it is important to provide a path for extension
types to have serialized metadata as binary because the alternatives are
(in my opinion) not great and might lead to extension developers inventing
their own key/value encoding.

With binary on the table there are options like protobuf, flatbuffers, or
memcpy of a struct that extension type implementations can use depending on
the complexity of the extension metadata that needs to be communicated and
whether or not the metadata is intended to be used in compiled code (i.e.,
are those values needed to do anything useful with the bytes in the
buffers?).

Without binary on the table, an extension type implementation has to vendor
in a JSON reader/writer, vendor in a base64 encoder/decoder, or invent some
custom serialization that's easier to parse than either of those.

Cheers,

-dewey

On Fri, Feb 4, 2022 at 7:06 AM Joris Van den Bossche <
jorisvandenbossche@gmail.com> wrote:

> Reviving this thread, I don't think anything happened in the meantime on
> this topic?
>
> From rereading the thread, it seems David mentioned two possible ideas:
> - A new [byte] binary_value field in the existing KeyValue type, next to
> the existing string value field. And if you have valid string metadata, the
> same value can be put in both fields by reusing the offset
> - Relax the existing string value field to [byte], but also require
> implementations to null-terminate binary values regardless to ensure
> compatibility for old readers
>  - And in addition to those options, there is also the safest option of
> adding a new BinaryKeyValue type and add that next to KeyValue everywhere
> this is used.
>
> And we need to further assess which option is viable given the
> forward/backward compatibility constraints?
>
> I want to point out though that in pyarrow we already _do_ use binary
> metadata for extension types for several years. The built-in
> PyExtensionType class uses a pickle dump as the serialized metadata (so
> which is put in the ARROW:extension:metadata key in the key-value metadata
> of the Field), and the bytes from a pickle dump are certainly not valid
> UTF-8 I think.
> I don't know how much this is used in the wild. In pandas, our arrow
> extension types don't use this pickle-based PyExtensionType but define
> their own serialization which currently returns a valid UTF-8 string. But
> for example the Ray project seems to use PyExtensionType (
>
> https://docs.ray.io/en/latest/_modules/ray/data/extensions/tensor_extension.html#ArrowTensorType
> ).
>
> Joris
>
> On Thu, 2 Sept 2021 at 07:01, Micah Kornfield <em...@gmail.com>
> wrote:
>
> > It still sounds like adding a new type might be the safest approach (and
> > marking the old type as discouraged).
> >
> > On Mon, Aug 23, 2021 at 11:18 AM David Li <li...@apache.org> wrote:
> >
> > > I believe so.
> > >
> > > The encoding of a string in Flatbuffers is [byte] with a null
> terminator
> > > not included in the length, so old files should still be readable (they
> > > would simply not see the terminator anymore). And conversely,
> continuing
> > to
> > > write the null terminator means new files should still be readable by
> > > existing implementations, so long as they don't have binary metadata
> (but
> > > we could increment the metadata version before allowing that). There
> > still
> > > remains the issue of non-UTF8 data inside the "string" value; on that
> > > point, this is a non-answer (or I guess, it canonicalizes the current
> > > unintentional behavior).
> > >
> > > It would complicate the Java API as that currently works with Strings,
> > but
> > > that will be an issue for any attempt to add binary metadata.
> > >
> > > -David
> > >
> > > On Mon, Aug 23, 2021, at 13:40, Antoine Pitrou wrote:
> > > >
> > > > Le 23/08/2021 à 17:52, David Li a écrit :
> > > > > Another way forward might be to relax the value type to [byte], but
> > > also require implementations to null-terminate binary values
> regardless.
> > > The C++ Flatbuffers implementation does this already [1] (though not
> the
> > > Java one [2]). Old implementations validating UTF8-ness would still be
> > > unable to read anything with binary metadata (which is the case they're
> > > already in), but implementations just validating for null-termination
> > would
> > > continue to work. And again, I believe this is in-spec for Flatbuffers
> > > since string lengths don't include the null terminator.
> > > > >
> > > > > [1]:
> > >
> >
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670
> > > > > [2]:
> > >
> >
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637
> > > > > Though it seems we could emulate C++ here by calling addByte(0)
> > > ourselves before createByteVector.
> > > >
> > > > Would that allow for both forwards and backwards compatibility when
> the
> > > > metadata is valid UTF8?
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Joris Van den Bossche <jo...@gmail.com>]

Reviving this thread, I don't think anything happened in the meantime on
this topic?

From rereading the thread, it seems David mentioned two possible ideas:
- A new [byte] binary_value field in the existing KeyValue type, next to
the existing string value field. And if you have valid string metadata, the
same value can be put in both fields by reusing the offset
- Relax the existing string value field to [byte], but also require
implementations to null-terminate binary values regardless to ensure
compatibility for old readers
 - And in addition to those options, there is also the safest option of
adding a new BinaryKeyValue type and add that next to KeyValue everywhere
this is used.

And we need to further assess which option is viable given the
forward/backward compatibility constraints?

I want to point out though that in pyarrow we already _do_ use binary
metadata for extension types for several years. The built-in
PyExtensionType class uses a pickle dump as the serialized metadata (so
which is put in the ARROW:extension:metadata key in the key-value metadata
of the Field), and the bytes from a pickle dump are certainly not valid
UTF-8 I think.
I don't know how much this is used in the wild. In pandas, our arrow
extension types don't use this pickle-based PyExtensionType but define
their own serialization which currently returns a valid UTF-8 string. But
for example the Ray project seems to use PyExtensionType (
https://docs.ray.io/en/latest/_modules/ray/data/extensions/tensor_extension.html#ArrowTensorType
).

Joris

On Thu, 2 Sept 2021 at 07:01, Micah Kornfield <em...@gmail.com> wrote:

> It still sounds like adding a new type might be the safest approach (and
> marking the old type as discouraged).
>
> On Mon, Aug 23, 2021 at 11:18 AM David Li <li...@apache.org> wrote:
>
> > I believe so.
> >
> > The encoding of a string in Flatbuffers is [byte] with a null terminator
> > not included in the length, so old files should still be readable (they
> > would simply not see the terminator anymore). And conversely, continuing
> to
> > write the null terminator means new files should still be readable by
> > existing implementations, so long as they don't have binary metadata (but
> > we could increment the metadata version before allowing that). There
> still
> > remains the issue of non-UTF8 data inside the "string" value; on that
> > point, this is a non-answer (or I guess, it canonicalizes the current
> > unintentional behavior).
> >
> > It would complicate the Java API as that currently works with Strings,
> but
> > that will be an issue for any attempt to add binary metadata.
> >
> > -David
> >
> > On Mon, Aug 23, 2021, at 13:40, Antoine Pitrou wrote:
> > >
> > > Le 23/08/2021 à 17:52, David Li a écrit :
> > > > Another way forward might be to relax the value type to [byte], but
> > also require implementations to null-terminate binary values regardless.
> > The C++ Flatbuffers implementation does this already [1] (though not the
> > Java one [2]). Old implementations validating UTF8-ness would still be
> > unable to read anything with binary metadata (which is the case they're
> > already in), but implementations just validating for null-termination
> would
> > continue to work. And again, I believe this is in-spec for Flatbuffers
> > since string lengths don't include the null terminator.
> > > >
> > > > [1]:
> >
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670
> > > > [2]:
> >
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637
> > > > Though it seems we could emulate C++ here by calling addByte(0)
> > ourselves before createByteVector.
> > >
> > > Would that allow for both forwards and backwards compatibility when the
> > > metadata is valid UTF8?
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> >
>

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

It still sounds like adding a new type might be the safest approach (and
marking the old type as discouraged).

On Mon, Aug 23, 2021 at 11:18 AM David Li <li...@apache.org> wrote:

> I believe so.
>
> The encoding of a string in Flatbuffers is [byte] with a null terminator
> not included in the length, so old files should still be readable (they
> would simply not see the terminator anymore). And conversely, continuing to
> write the null terminator means new files should still be readable by
> existing implementations, so long as they don't have binary metadata (but
> we could increment the metadata version before allowing that). There still
> remains the issue of non-UTF8 data inside the "string" value; on that
> point, this is a non-answer (or I guess, it canonicalizes the current
> unintentional behavior).
>
> It would complicate the Java API as that currently works with Strings, but
> that will be an issue for any attempt to add binary metadata.
>
> -David
>
> On Mon, Aug 23, 2021, at 13:40, Antoine Pitrou wrote:
> >
> > Le 23/08/2021 à 17:52, David Li a écrit :
> > > Another way forward might be to relax the value type to [byte], but
> also require implementations to null-terminate binary values regardless.
> The C++ Flatbuffers implementation does this already [1] (though not the
> Java one [2]). Old implementations validating UTF8-ness would still be
> unable to read anything with binary metadata (which is the case they're
> already in), but implementations just validating for null-termination would
> continue to work. And again, I believe this is in-spec for Flatbuffers
> since string lengths don't include the null terminator.
> > >
> > > [1]:
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670
> > > [2]:
> https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637
> > > Though it seems we could emulate C++ here by calling addByte(0)
> ourselves before createByteVector.
> >
> > Would that allow for both forwards and backwards compatibility when the
> > metadata is valid UTF8?
> >
> > Regards
> >
> > Antoine.
> >
> >
>

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

I believe so.

The encoding of a string in Flatbuffers is [byte] with a null terminator not included in the length, so old files should still be readable (they would simply not see the terminator anymore). And conversely, continuing to write the null terminator means new files should still be readable by existing implementations, so long as they don't have binary metadata (but we could increment the metadata version before allowing that). There still remains the issue of non-UTF8 data inside the "string" value; on that point, this is a non-answer (or I guess, it canonicalizes the current unintentional behavior).

It would complicate the Java API as that currently works with Strings, but that will be an issue for any attempt to add binary metadata.

-David

On Mon, Aug 23, 2021, at 13:40, Antoine Pitrou wrote:
> 
> Le 23/08/2021 à 17:52, David Li a écrit :
> > Another way forward might be to relax the value type to [byte], but also require implementations to null-terminate binary values regardless. The C++ Flatbuffers implementation does this already [1] (though not the Java one [2]). Old implementations validating UTF8-ness would still be unable to read anything with binary metadata (which is the case they're already in), but implementations just validating for null-termination would continue to work. And again, I believe this is in-spec for Flatbuffers since string lengths don't include the null terminator.
> > 
> > [1]: https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670
> > [2]: https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637
> > Though it seems we could emulate C++ here by calling addByte(0) ourselves before createByteVector.
> 
> Would that allow for both forwards and backwards compatibility when the 
> metadata is valid UTF8?
> 
> Regards
> 
> Antoine.
> 
> 

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Antoine Pitrou <an...@python.org>]

Le 23/08/2021 à 17:52, David Li a écrit :
> Another way forward might be to relax the value type to [byte], but also require implementations to null-terminate binary values regardless. The C++ Flatbuffers implementation does this already [1] (though not the Java one [2]). Old implementations validating UTF8-ness would still be unable to read anything with binary metadata (which is the case they're already in), but implementations just validating for null-termination would continue to work. And again, I believe this is in-spec for Flatbuffers since string lengths don't include the null terminator.
> 
> [1]: https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670
> [2]: https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637
> Though it seems we could emulate C++ here by calling addByte(0) ourselves before createByteVector.

Would that allow for both forwards and backwards compatibility when the 
metadata is valid UTF8?

Regards

Antoine.

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Another way forward might be to relax the value type to [byte], but also require implementations to null-terminate binary values regardless. The C++ Flatbuffers implementation does this already [1] (though not the Java one [2]). Old implementations validating UTF8-ness would still be unable to read anything with binary metadata (which is the case they're already in), but implementations just validating for null-termination would continue to work. And again, I believe this is in-spec for Flatbuffers since string lengths don't include the null terminator.

[1]: https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/include/flatbuffers/flatbuffers.h#L1612-L1670
[2]: https://github.com/google/flatbuffers/blob/273f6084e55285e26ff8b4cdd55a9c0e2d9a48b7/java/com/google/flatbuffers/FlatBufferBuilder.java#L594-L637
Though it seems we could emulate C++ here by calling addByte(0) ourselves before createByteVector.

-David

On Wed, Aug 18, 2021, at 13:20, David Li wrote:
> This isn't too thought out yet but:
> 1.  Any file which stuffs binary data into the value is already unreadable for anyone directly using Flatbuffers. So we can specify that the field must be valid UTF-8, but implementations can permit relaxed validation/reading as binary data instead in order to read old files for the current metadata version.
> 2. For the new version, we can add a [byte] field to KeyValueMetadata alongside the string field. Since strings in Flatbuffers don't include the null terminator in their length, implementations can reuse the offset to allow values which are valid UTF-8 to be placed in both fields at a cost of only 4 bytes, so old consumers can still see UTF-8 metadata values. Actual binary values would only exist in the new field, and old consumers would see a null value (since we didn't declare it required); if both fields have a value, the binary value takes precedence. I think this at least leaves our read APIs alone since they already use bytestrings.
> A current extension type that uses binary metadata would still be a problem, though…
> 
> -David
> 
> On Tue, Aug 17, 2021, at 00:08, Micah Kornfield wrote:
> > I agree with you any thoughts on a way forward for at least hardening the
> > spec (or should this be done at the same time as adding the new field)?
> > 
> > On Mon, Aug 16, 2021 at 1:45 AM Wes McKinney <we...@gmail.com> wrote:
> > 
> > > I've been poking around the project, and I'm growing concerned that
> > > our use of the KeyValue field has already been non-compliant in many
> > > cases since we do not validate UTF8-ness. Since we also use KeyValue
> > > to handle opaque data serialization for extension types [1], the fact
> > > that the specification does not clarify that binary data (such as the
> > > output of ExtensionType::Serialize) must be base64-encoded or similar
> > > makes this a bit of a minefield at the moment.
> > >
> > > It seems that there are no particularly excellent solutions, and
> > > maintaining the status quo (having now identified these
> > > inconsistencies / vagueries in at least the C++ implementation) is
> > > probably not a good idea either. In Parquet, where we store a
> > > serialized binary Arrow schema, we have to base64-encode [2]
> > >
> > > [1]:
> > > https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
> > > [2]:
> > > https://github.com/apache/arrow/blob/e990d177b1f1dec962315487682f613d46be573c/cpp/src/parquet/arrow/writer.cc#L423
> > >
> > > On Tue, Aug 10, 2021 at 3:27 PM Wes McKinney <we...@gmail.com> wrote:
> > > >
> > > > Ah, that's definitely a no-go then (I believe we verify messages
> > > > unconditionally in C++). That's unfortunate (and I feel responsible
> > > > for missing this, but I suppose we had a lot of opportunities to fix
> > > > it prior to the 1.0.0 format version) — so to have actual binary
> > > > values (which was the intention in the first place for the metadata)
> > > > we would need to add a new metadata field.
> > > >
> > > > On Tue, Aug 10, 2021 at 6:53 AM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > > > >
> > > > > One issue with changing it to byte is it would effectively break any
> > > reader that is validating flatbuffer data, because flatbuffers verifies
> > > null termination [1].  While this might comply with forward compatibility
> > > guarantees it seems like a pretty large blast radius.
> > > > >
> > > > > [1]
> > > https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flatbuffers.h#L2457
> > > > >
> > > > > On Mon, Jul 12, 2021 at 12:38 PM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> pyarrow at least treats the KeyValue values as binary and not UTF-8.
> > > > >>
> > > > >> On Sun, Jul 11, 2021 at 9:40 PM Micah Kornfield <
> > > emkornfield@gmail.com> wrote:
> > > > >> >
> > > > >> > I think other languages (e.g. java, python) might make more of
> > > distinction between utf-8 compatible strings and raw bytes.  For python it
> > > might be less of a concern if the c++ wrapper already makes the value field
> > > look like a bytes field
> > > > >> >
> > > > >> > On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:
> > > > >> >>
> > > > >> >> We could certainly "upgrade" KeyValue to have a binary value field
> > > > >> >> everywhere KeyValue is used, but there is some risk of code in the
> > > > >> >> wild expecting there to be a null terminator after the string data.
> > > > >> >> The Flatbuffers-generated accessor APIs do not depend on the
> > > existence
> > > > >> >> of the null terminator, though. Not ideal, but I would not be
> > > thrilled
> > > > >> >> about adding an extra [ BinaryKeyValue ] everyplace we currently
> > > have
> > > > >> >> [ KeyValue ].
> > > > >> >>
> > > > >> >> That said, I doubt that we have any endogenous forward
> > > compatibility
> > > > >> >> problems related to this in Apache Arrow-maintained libraries, the
> > > > >> >> risk would come from users who are interacting with the Flatbuffers
> > > > >> >> data manually / without using one of our libraries. We could
> > > implement
> > > > >> >> the changes and run a set of forward compatibility integration
> > > tests
> > > > >> >> to see if anyone of our released libraries have an issue.
> > > > >> >>
> > > > >> >> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com>
> > > wrote:
> > > > >> >> >
> > > > >> >> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
> > > > >> >> >
> > > > >> >> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <
> > > emkornfield@gmail.com> wrote:
> > > > >> >> > >
> > > > >> >> > > Retitling and forking the discussion to talk about key value
> > > pairs.
> > > > >> >> > >
> > > > >> >> > > What is the byte cost of an empty list?  Another option would
> > > be to
> > > > >> >> > > introduce a new BinaryKeyValue table and add binary metadata.
> > > > >> >> > >
> > > > >> >> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > > > >> >> > > natebauernfeind@deephaven.io> wrote:
> > > > >> >> > >
> > > > >> >> > > > Deephaven and I are very supportive of "upgrading" the value
> > > half of the kv
> > > > >> >> > > > pair to a byte vector. What is the best way to find out if
> > > there is
> > > > >> >> > > > sufficient interest?
> > > > >> >> > > >
> > > > >> >> > > >
> > > > >> >> > > > I've been stewing on the ideas here around schema evolution,
> > > and I realize
> > > > >> >> > > > the specific feature I am missing is the ability to encode
> > > that a field
> > > > >> >> > > > (i.e. its FieldNode and accompanying Buffers in the
> > > RecordBatch) is
> > > > >> >> > > > empty/has-no-data in O(0) cost (yes; for free).
> > > > >> >> > > >
> > > > >> >> > > > Might there be interest in adding a "field_id" to the
> > > FieldNode (which is
> > > > >> >> > > > encoded on the RecordBatch flatbuffer)? I see a simple
> > > forward-compatible
> > > > >> >> > > > upgrade (by either keying off of 0, or explicitly set the
> > > field default to
> > > > >> >> > > > -1) which would allow the sender to "skip" fields that have
> > > 1) FieldNode
> > > > >> >> > > > length of zero, and 2) all Buffer's associated at that level
> > > (and further
> > > > >> >> > > > nested) are also equally empty (i.e. Buffer length is zero).
> > > > >> >> > > >
> > > > >> >> > > > I understand this concept slightly interferes with
> > > RecordBatch's `length`
> > > > >> >> > > > field, and that many implementations use that length to
> > > resize the
> > > > >> >> > > > root-level FieldNodes. The use-case I have in mind has
> > > different logical
> > > > >> >> > > > lengths per field node; current implementations require
> > > sending a
> > > > >> >> > > > RecordBatch length of the max length across all root level
> > > field nodes. I
> > > > >> >> > > > believe this requires a copy of data whenever a field node
> > > is too short; I
> > > > >> >> > > > don't know if there is a decent solution to this slight
> > > inefficiency. I am
> > > > >> >> > > > bringing it up because if "skipping a field node when it is
> > > empty" is a
> > > > >> >> > > > feature, then we may not want to allocate space for those
> > > nodes given that
> > > > >> >> > > > the record batch length will likely be greater than zero.
> > > > >> >> > > >
> > > > >> >> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <
> > > wesmckinn@gmail.com> wrote:
> > > > >> >> > > >
> > > > >> >> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <
> > > apache@lidavidm.me> wrote:
> > > > >> >> > > > > >
> > > > >> >> > > > > > From the Flatbuffers internals doc[1] it appears they
> > > are the same:
> > > > >> >> > > > > "Strings are simply a vector of bytes, and are always
> > > null-terminated."
> > > > >> >> > > > >
> > > > >> >> > > > > I see. I took a look at flatbuffers.h, and it appears that
> > > changing
> > > > >> >> > > > > this field from string to [byte] would be
> > > backward-compatible and
> > > > >> >> > > > > forward-compatible except with code that expects a null
> > > terminator.
> > > > >> >> > > > > This is something we could discuss separately if there
> > > were enough
> > > > >> >> > > > > interest.
> > > > >> >> > > > >
> > > > >> >> > > > > > [1]:
> > > https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > > >> >> > > > > >
> > > > >> >> > > > > > -David
> > > > >> >> > > > > >
> > > > >> >> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > >> >> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > > > >> >> > > > emkornfield@gmail.com>
> > > > >> >> > > > > wrote:
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > Right, I had wanted to focus the discussion on
> > > Flight as I think
> > > > >> >> > > > > schema
> > > > >> >> > > > > > > > > evolution or multiplexing streams (more so the
> > > latter) is a
> > > > >> >> > > > > property of the
> > > > >> >> > > > > > > > > transport and not the stream format itself. If we
> > > are leaning
> > > > >> >> > > > > towards just
> > > > >> >> > > > > > > > > schema evolution then maybe it makes sense to
> > > discuss it for the
> > > > >> >> > > > > IPC stream
> > > > >> >> > > > > > > > > format and leverage that in Flight. I'd be
> > > interested in what
> > > > >> >> > > > > others think.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > I tend to agree, I think stream multiplexing is
> > > likely a transport
> > > > >> >> > > > > level
> > > > >> >> > > > > > > > issue.  IMO I think schema evolution should be
> > > consistent with the
> > > > >> >> > > > > IPC
> > > > >> >> > > > > > > > stream format  and flight.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > > Nate: it may be worth starting a separate
> > > discussion about more
> > > > >> >> > > > > general
> > > > >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> > > key-value
> > > > >> >> > > > > metadata was
> > > > >> >> > > > > > > > > chosen/if opaque bytes were considered in the past.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > I think  this was an unfortunate design of the key
> > > value metadata
> > > > >> >> > > > in
> > > > >> >> > > > > > > > Schema.fbs, but I don't think I was around when this
> > > decision was
> > > > >> >> > > > > made.
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > I agree that it's unfortunate that we did not use [
> > > byte ] instead of
> > > > >> >> > > > > > > string for the value in the KeyValue metadata — I
> > > think this was more
> > > > >> >> > > > > > > of an oversight than a deliberate choice (e.g. it was
> > > not our intent
> > > > >> >> > > > > > > to require binary data to be base64-encoded — this is
> > > something that
> > > > >> >> > > > > > > we have to do when encoding binary data in Thrift
> > > KeyValue metadata
> > > > >> >> > > > > > > for Parquet, for example). Is the binary
> > > representation of [byte]
> > > > >> >> > > > > > > different from string?
> > > > >> >> > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > > > > > Side Question: Why isn't the IPC stream format a
> > > series of the
> > > > >> >> > > > flight
> > > > >> >> > > > > > > > > protobufs?
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > In addition to what David said, protobufs can't be
> > > read directly
> > > > >> >> > > > > from a
> > > > >> >> > > > > > > > memory-mapped file (they need decoding).  This was
> > > one of the
> > > > >> >> > > > design
> > > > >> >> > > > > > > > considerations of using flatbuffers and IPC
> > > Stream/File format.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > I was thinking Micah's comment is more that whatever
> > > we do, it
> > > > >> >> > > > > should be
> > > > >> >> > > > > > > > > clearly specified and edge cases should be
> > > considered, especially
> > > > >> >> > > > > if we
> > > > >> >> > > > > > > > > might want to 'backport' this into the stream
> > > format later.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > Yes, for dictionaries we just need to be careful to
> > > define
> > > > >> >> > > > semantics
> > > > >> >> > > > > and
> > > > >> >> > > > > > > > ensure implementations are validating them with
> > > regards to
> > > > >> >> > > > > dictionaries.
> > > > >> >> > > > > > > > There likely isn't any need to change current
> > > implementations
> > > > >> >> > > > though.
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <
> > > lidavidm@apache.org>
> > > > >> >> > > > > wrote:
> > > > >> >> > > > > > > >
> > > > >> >> > > > > > > > > Right, I had wanted to focus the discussion on
> > > Flight as I think
> > > > >> >> > > > > schema
> > > > >> >> > > > > > > > > evolution or multiplexing streams (more so the
> > > latter) is a
> > > > >> >> > > > > property of the
> > > > >> >> > > > > > > > > transport and not the stream format itself. If we
> > > are leaning
> > > > >> >> > > > > towards just
> > > > >> >> > > > > > > > > schema evolution then maybe it makes sense to
> > > discuss it for the
> > > > >> >> > > > > IPC stream
> > > > >> >> > > > > > > > > format and leverage that in Flight. I'd be
> > > interested in what
> > > > >> >> > > > > others think.
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > Especially if we are looking at multiplexing
> > > streams - I would
> > > > >> >> > > > > wonder if
> > > > >> >> > > > > > > > > that's actually better served by making it easier
> > > to implement
> > > > >> >> > > > > using the
> > > > >> >> > > > > > > > > Flight implementation as it stands (by managing
> > > concurrent RPC
> > > > >> >> > > > > calls and/or
> > > > >> >> > > > > > > > > performing the union-of-structs encoding trick for
> > > you), instead
> > > > >> >> > > > > of having
> > > > >> >> > > > > > > > > to change the protocol.
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > Nate: it may be worth starting a separate
> > > discussion about more
> > > > >> >> > > > > general
> > > > >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> > > key-value
> > > > >> >> > > > > metadata was
> > > > >> >> > > > > > > > > chosen/if opaque bytes were considered in the
> > > past. Off the top
> > > > >> >> > > > of
> > > > >> >> > > > > my head
> > > > >> >> > > > > > > > > if it's for on-disk storage and fully
> > > application-defined it may
> > > > >> >> > > > > make sense
> > > > >> >> > > > > > > > > to store as a separate file alongside the Arrow
> > > file (indexed by
> > > > >> >> > > > > record
> > > > >> >> > > > > > > > > batch index) where you can take advantage of
> > > whatever format is
> > > > >> >> > > > > most
> > > > >> >> > > > > > > > > suitable.
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > -David
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan
> > > wrote:
> > > > >> >> > > > > > > > > > Hi guys,
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial
> > > suggestion was
> > > > >> >> > > > to
> > > > >> >> > > > > add this
> > > > >> >> > > > > > > > > > feature starting from the IPC(I moved initial
> > > write up steps to
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > > bottom
> > > > >> >> > > > > > > > > > of the doc). Afterwards David suggested focusing
> > > on Flight and
> > > > >> >> > > > > that's how
> > > > >> >> > > > > > > > > > we ended up with the protobufs change in the
> > > proposal. This
> > > > >> >> > > > > being said I
> > > > >> >> > > > > > > > > do
> > > > >> >> > > > > > > > > > think that the place where this should be
> > > impemented is a good
> > > > >> >> > > > > question
> > > > >> >> > > > > > > > > on
> > > > >> >> > > > > > > > > > its own. Maybe it makes sense to have this kind
> > > of a feature in
> > > > >> >> > > > > IPC and
> > > > >> >> > > > > > > > > > somehow use it in Flight, maybe not.
> > > > >> >> > > > > > > > > > 2. The point about dictionaries deserves a
> > > dedicated section in
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > > > proposal. Nate and David brought it up and
> > > shared some
> > > > >> >> > > > insights.
> > > > >> >> > > > > I'll try
> > > > >> >> > > > > > > > > > to aggregate them and we can continue the
> > > discussion form
> > > > >> >> > > > there.
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > Cheers,
> > > > >> >> > > > > > > > > > Gosh
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > >> >> > > > > > > > > natebauernfeind@deephaven.io>
> > > > >> >> > > > > > > > > > wrote:
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > > > > >
> > > > >> >> > > > > > > > > > > > > > makes it more difficult to bring schema
> > > evolution back
> > > > >> >> > > > > into the
> > > > >> >> > > > > > > > > > > > > > IPC Stream format (i.e. it would live
> > > only in flight)
> > > > >> >> > > > > > > > > > > > >
> > > > >> >> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer
> > > structures not the
> > > > >> >> > > > > > > > > protobufs.
> > > > >> >> > > > > > > > > > > Can
> > > > >> >> > > > > > > > > > > > > you help me understand how difficult it
> > > would be to bring
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > > > > `schema_id`
> > > > >> >> > > > > > > > > > > > > approach to the IPC stream format?
> > > > >> >> > > > > > > > > > > >
> > > > >> >> > > > > > > > > > > > I thought we were talking solely about the
> > > Flight Protobuf
> > > > >> >> > > > > > > > > definitions -
> > > > >> >> > > > > > > > > > > > not the Flatbuffers (and the Google doc at
> > > least only talks
> > > > >> >> > > > > about the
> > > > >> >> > > > > > > > > > > > Protobufs).
> > > > >> >> > > > > > > > > > > >
> > > > >> >> > > > > > > > > > >
> > > > >> >> > > > > > > > > > > I somehow missed that schema_id is being added
> > > to protobuf in
> > > > >> >> > > > > the
> > > > >> >> > > > > > > > > document.
> > > > >> >> > > > > > > > > > > It feels to me that the schema_id is a
> > > property that would
> > > > >> >> > > > > ideally only
> > > > >> >> > > > > > > > > > > apply to the RecordBatch. I better understand
> > > Micah's
> > > > >> >> > > > > dictionary
> > > > >> >> > > > > > > > > concerns,
> > > > >> >> > > > > > > > > > > now, too.
> > > > >> >> > > > > > > > > > >
> > > > >> >> > > > > > > > > > > > Side Question: Why isn't the IPC stream
> > > format a series of
> > > > >> >> > > > > the flight
> > > > >> >> > > > > > > > > > > > > protobufs? It's a real shame that there is
> > > no standard
> > > > >> >> > > > way
> > > > >> >> > > > > to
> > > > >> >> > > > > > > > > > > > > capture/replay a stream with app_metadata.
> > > (Obviously
> > > > >> >> > > > > ignoring the
> > > > >> >> > > > > > > > > > > > > annoyances around protobuf wrapping
> > > flatbuffers.)
> > > > >> >> > > > > > > > > > > >
> > > > >> >> > > > > > > > > > > > The IPC format was defined long before
> > > Flight, and Flight's
> > > > >> >> > > > > > > > > app_metadata
> > > > >> >> > > > > > > > > > > > was added after Flight's initial definition.
> > > Note an IPC
> > > > >> >> > > > > message does
> > > > >> >> > > > > > > > > > > have
> > > > >> >> > > > > > > > > > > > a provision for key-value metadata, though I
> > > think APIs for
> > > > >> >> > > > > that are
> > > > >> >> > > > > > > > > not
> > > > >> >> > > > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > >> >> > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > > >> >> > > > > despite my
> > > > >> >> > > > > > > > > comments
> > > > >> >> > > > > > > > > > > > there perhaps we need to unify or at least
> > > consider how
> > > > >> >> > > > > Flight's
> > > > >> >> > > > > > > > > > > > app_metadata relates to the IPC message
> > > custom_metadata.
> > > > >> >> > > > Also
> > > > >> >> > > > > > > > > perhaps see
> > > > >> >> > > > > > > > > > > > ARROW-1059.)
> > > > >> >> > > > > > > > > > > >
> > > > >> >> > > > > > > > > > >
> > > > >> >> > > > > > > > > > > KeyValue unfortunately is string to string. In
> > > flatbuffer
> > > > >> >> > > > > strings are
> > > > >> >> > > > > > > > > only
> > > > >> >> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the
> > > other hand is
> > > > >> >> > > > > opaque
> > > > >> >> > > > > > > > > bytes.
> > > > >> >> > > > > > > > > > > The latter is a bit more useful.
> > > > >> >> > > > > > > > > > >
> > > > >> >> > > > > > > > > > > --
> > > > >> >> > > > > > > > > > >
> > > > >> >> > > > > > > > > >
> > > > >> >> > > > > > > > >
> > > > >> >> > > > > > >
> > > > >> >> > > > >
> > > > >> >> > > >
> > > > >> >> > > >
> > > > >> >> > > > --
> > > > >> >> > > >
> > >
> > 
> 

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

This isn't too thought out yet but:
 1.  Any file which stuffs binary data into the value is already unreadable for anyone directly using Flatbuffers. So we can specify that the field must be valid UTF-8, but implementations can permit relaxed validation/reading as binary data instead in order to read old files for the current metadata version.
 2. For the new version, we can add a [byte] field to KeyValueMetadata alongside the string field. Since strings in Flatbuffers don't include the null terminator in their length, implementations can reuse the offset to allow values which are valid UTF-8 to be placed in both fields at a cost of only 4 bytes, so old consumers can still see UTF-8 metadata values. Actual binary values would only exist in the new field, and old consumers would see a null value (since we didn't declare it required); if both fields have a value, the binary value takes precedence. I think this at least leaves our read APIs alone since they already use bytestrings.
A current extension type that uses binary metadata would still be a problem, though…

-David

On Tue, Aug 17, 2021, at 00:08, Micah Kornfield wrote:
> I agree with you any thoughts on a way forward for at least hardening the
> spec (or should this be done at the same time as adding the new field)?
> 
> On Mon, Aug 16, 2021 at 1:45 AM Wes McKinney <we...@gmail.com> wrote:
> 
> > I've been poking around the project, and I'm growing concerned that
> > our use of the KeyValue field has already been non-compliant in many
> > cases since we do not validate UTF8-ness. Since we also use KeyValue
> > to handle opaque data serialization for extension types [1], the fact
> > that the specification does not clarify that binary data (such as the
> > output of ExtensionType::Serialize) must be base64-encoded or similar
> > makes this a bit of a minefield at the moment.
> >
> > It seems that there are no particularly excellent solutions, and
> > maintaining the status quo (having now identified these
> > inconsistencies / vagueries in at least the C++ implementation) is
> > probably not a good idea either. In Parquet, where we store a
> > serialized binary Arrow schema, we have to base64-encode [2]
> >
> > [1]:
> > https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
> > [2]:
> > https://github.com/apache/arrow/blob/e990d177b1f1dec962315487682f613d46be573c/cpp/src/parquet/arrow/writer.cc#L423
> >
> > On Tue, Aug 10, 2021 at 3:27 PM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > Ah, that's definitely a no-go then (I believe we verify messages
> > > unconditionally in C++). That's unfortunate (and I feel responsible
> > > for missing this, but I suppose we had a lot of opportunities to fix
> > > it prior to the 1.0.0 format version) — so to have actual binary
> > > values (which was the intention in the first place for the metadata)
> > > we would need to add a new metadata field.
> > >
> > > On Tue, Aug 10, 2021 at 6:53 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> > > >
> > > > One issue with changing it to byte is it would effectively break any
> > reader that is validating flatbuffer data, because flatbuffers verifies
> > null termination [1].  While this might comply with forward compatibility
> > guarantees it seems like a pretty large blast radius.
> > > >
> > > > [1]
> > https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flatbuffers.h#L2457
> > > >
> > > > On Mon, Jul 12, 2021 at 12:38 PM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >>
> > > >> pyarrow at least treats the KeyValue values as binary and not UTF-8.
> > > >>
> > > >> On Sun, Jul 11, 2021 at 9:40 PM Micah Kornfield <
> > emkornfield@gmail.com> wrote:
> > > >> >
> > > >> > I think other languages (e.g. java, python) might make more of
> > distinction between utf-8 compatible strings and raw bytes.  For python it
> > might be less of a concern if the c++ wrapper already makes the value field
> > look like a bytes field
> > > >> >
> > > >> > On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:
> > > >> >>
> > > >> >> We could certainly "upgrade" KeyValue to have a binary value field
> > > >> >> everywhere KeyValue is used, but there is some risk of code in the
> > > >> >> wild expecting there to be a null terminator after the string data.
> > > >> >> The Flatbuffers-generated accessor APIs do not depend on the
> > existence
> > > >> >> of the null terminator, though. Not ideal, but I would not be
> > thrilled
> > > >> >> about adding an extra [ BinaryKeyValue ] everyplace we currently
> > have
> > > >> >> [ KeyValue ].
> > > >> >>
> > > >> >> That said, I doubt that we have any endogenous forward
> > compatibility
> > > >> >> problems related to this in Apache Arrow-maintained libraries, the
> > > >> >> risk would come from users who are interacting with the Flatbuffers
> > > >> >> data manually / without using one of our libraries. We could
> > implement
> > > >> >> the changes and run a set of forward compatibility integration
> > tests
> > > >> >> to see if anyone of our released libraries have an issue.
> > > >> >>
> > > >> >> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com>
> > wrote:
> > > >> >> >
> > > >> >> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
> > > >> >> >
> > > >> >> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <
> > emkornfield@gmail.com> wrote:
> > > >> >> > >
> > > >> >> > > Retitling and forking the discussion to talk about key value
> > pairs.
> > > >> >> > >
> > > >> >> > > What is the byte cost of an empty list?  Another option would
> > be to
> > > >> >> > > introduce a new BinaryKeyValue table and add binary metadata.
> > > >> >> > >
> > > >> >> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > > >> >> > > natebauernfeind@deephaven.io> wrote:
> > > >> >> > >
> > > >> >> > > > Deephaven and I are very supportive of "upgrading" the value
> > half of the kv
> > > >> >> > > > pair to a byte vector. What is the best way to find out if
> > there is
> > > >> >> > > > sufficient interest?
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > > I've been stewing on the ideas here around schema evolution,
> > and I realize
> > > >> >> > > > the specific feature I am missing is the ability to encode
> > that a field
> > > >> >> > > > (i.e. its FieldNode and accompanying Buffers in the
> > RecordBatch) is
> > > >> >> > > > empty/has-no-data in O(0) cost (yes; for free).
> > > >> >> > > >
> > > >> >> > > > Might there be interest in adding a "field_id" to the
> > FieldNode (which is
> > > >> >> > > > encoded on the RecordBatch flatbuffer)? I see a simple
> > forward-compatible
> > > >> >> > > > upgrade (by either keying off of 0, or explicitly set the
> > field default to
> > > >> >> > > > -1) which would allow the sender to "skip" fields that have
> > 1) FieldNode
> > > >> >> > > > length of zero, and 2) all Buffer's associated at that level
> > (and further
> > > >> >> > > > nested) are also equally empty (i.e. Buffer length is zero).
> > > >> >> > > >
> > > >> >> > > > I understand this concept slightly interferes with
> > RecordBatch's `length`
> > > >> >> > > > field, and that many implementations use that length to
> > resize the
> > > >> >> > > > root-level FieldNodes. The use-case I have in mind has
> > different logical
> > > >> >> > > > lengths per field node; current implementations require
> > sending a
> > > >> >> > > > RecordBatch length of the max length across all root level
> > field nodes. I
> > > >> >> > > > believe this requires a copy of data whenever a field node
> > is too short; I
> > > >> >> > > > don't know if there is a decent solution to this slight
> > inefficiency. I am
> > > >> >> > > > bringing it up because if "skipping a field node when it is
> > empty" is a
> > > >> >> > > > feature, then we may not want to allocate space for those
> > nodes given that
> > > >> >> > > > the record batch length will likely be greater than zero.
> > > >> >> > > >
> > > >> >> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <
> > wesmckinn@gmail.com> wrote:
> > > >> >> > > >
> > > >> >> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <
> > apache@lidavidm.me> wrote:
> > > >> >> > > > > >
> > > >> >> > > > > > From the Flatbuffers internals doc[1] it appears they
> > are the same:
> > > >> >> > > > > "Strings are simply a vector of bytes, and are always
> > null-terminated."
> > > >> >> > > > >
> > > >> >> > > > > I see. I took a look at flatbuffers.h, and it appears that
> > changing
> > > >> >> > > > > this field from string to [byte] would be
> > backward-compatible and
> > > >> >> > > > > forward-compatible except with code that expects a null
> > terminator.
> > > >> >> > > > > This is something we could discuss separately if there
> > were enough
> > > >> >> > > > > interest.
> > > >> >> > > > >
> > > >> >> > > > > > [1]:
> > https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > >> >> > > > > >
> > > >> >> > > > > > -David
> > > >> >> > > > > >
> > > >> >> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > >> >> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > > >> >> > > > emkornfield@gmail.com>
> > > >> >> > > > > wrote:
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Right, I had wanted to focus the discussion on
> > Flight as I think
> > > >> >> > > > > schema
> > > >> >> > > > > > > > > evolution or multiplexing streams (more so the
> > latter) is a
> > > >> >> > > > > property of the
> > > >> >> > > > > > > > > transport and not the stream format itself. If we
> > are leaning
> > > >> >> > > > > towards just
> > > >> >> > > > > > > > > schema evolution then maybe it makes sense to
> > discuss it for the
> > > >> >> > > > > IPC stream
> > > >> >> > > > > > > > > format and leverage that in Flight. I'd be
> > interested in what
> > > >> >> > > > > others think.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > I tend to agree, I think stream multiplexing is
> > likely a transport
> > > >> >> > > > > level
> > > >> >> > > > > > > > issue.  IMO I think schema evolution should be
> > consistent with the
> > > >> >> > > > > IPC
> > > >> >> > > > > > > > stream format  and flight.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > > Nate: it may be worth starting a separate
> > discussion about more
> > > >> >> > > > > general
> > > >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> > key-value
> > > >> >> > > > > metadata was
> > > >> >> > > > > > > > > chosen/if opaque bytes were considered in the past.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > I think  this was an unfortunate design of the key
> > value metadata
> > > >> >> > > > in
> > > >> >> > > > > > > > Schema.fbs, but I don't think I was around when this
> > decision was
> > > >> >> > > > > made.
> > > >> >> > > > > > >
> > > >> >> > > > > > > I agree that it's unfortunate that we did not use [
> > byte ] instead of
> > > >> >> > > > > > > string for the value in the KeyValue metadata — I
> > think this was more
> > > >> >> > > > > > > of an oversight than a deliberate choice (e.g. it was
> > not our intent
> > > >> >> > > > > > > to require binary data to be base64-encoded — this is
> > something that
> > > >> >> > > > > > > we have to do when encoding binary data in Thrift
> > KeyValue metadata
> > > >> >> > > > > > > for Parquet, for example). Is the binary
> > representation of [byte]
> > > >> >> > > > > > > different from string?
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > > > > > Side Question: Why isn't the IPC stream format a
> > series of the
> > > >> >> > > > flight
> > > >> >> > > > > > > > > protobufs?
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > In addition to what David said, protobufs can't be
> > read directly
> > > >> >> > > > > from a
> > > >> >> > > > > > > > memory-mapped file (they need decoding).  This was
> > one of the
> > > >> >> > > > design
> > > >> >> > > > > > > > considerations of using flatbuffers and IPC
> > Stream/File format.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > I was thinking Micah's comment is more that whatever
> > we do, it
> > > >> >> > > > > should be
> > > >> >> > > > > > > > > clearly specified and edge cases should be
> > considered, especially
> > > >> >> > > > > if we
> > > >> >> > > > > > > > > might want to 'backport' this into the stream
> > format later.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > Yes, for dictionaries we just need to be careful to
> > define
> > > >> >> > > > semantics
> > > >> >> > > > > and
> > > >> >> > > > > > > > ensure implementations are validating them with
> > regards to
> > > >> >> > > > > dictionaries.
> > > >> >> > > > > > > > There likely isn't any need to change current
> > implementations
> > > >> >> > > > though.
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <
> > lidavidm@apache.org>
> > > >> >> > > > > wrote:
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > > Right, I had wanted to focus the discussion on
> > Flight as I think
> > > >> >> > > > > schema
> > > >> >> > > > > > > > > evolution or multiplexing streams (more so the
> > latter) is a
> > > >> >> > > > > property of the
> > > >> >> > > > > > > > > transport and not the stream format itself. If we
> > are leaning
> > > >> >> > > > > towards just
> > > >> >> > > > > > > > > schema evolution then maybe it makes sense to
> > discuss it for the
> > > >> >> > > > > IPC stream
> > > >> >> > > > > > > > > format and leverage that in Flight. I'd be
> > interested in what
> > > >> >> > > > > others think.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Especially if we are looking at multiplexing
> > streams - I would
> > > >> >> > > > > wonder if
> > > >> >> > > > > > > > > that's actually better served by making it easier
> > to implement
> > > >> >> > > > > using the
> > > >> >> > > > > > > > > Flight implementation as it stands (by managing
> > concurrent RPC
> > > >> >> > > > > calls and/or
> > > >> >> > > > > > > > > performing the union-of-structs encoding trick for
> > you), instead
> > > >> >> > > > > of having
> > > >> >> > > > > > > > > to change the protocol.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > Nate: it may be worth starting a separate
> > discussion about more
> > > >> >> > > > > general
> > > >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> > key-value
> > > >> >> > > > > metadata was
> > > >> >> > > > > > > > > chosen/if opaque bytes were considered in the
> > past. Off the top
> > > >> >> > > > of
> > > >> >> > > > > my head
> > > >> >> > > > > > > > > if it's for on-disk storage and fully
> > application-defined it may
> > > >> >> > > > > make sense
> > > >> >> > > > > > > > > to store as a separate file alongside the Arrow
> > file (indexed by
> > > >> >> > > > > record
> > > >> >> > > > > > > > > batch index) where you can take advantage of
> > whatever format is
> > > >> >> > > > > most
> > > >> >> > > > > > > > > suitable.
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > -David
> > > >> >> > > > > > > > >
> > > >> >> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan
> > wrote:
> > > >> >> > > > > > > > > > Hi guys,
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial
> > suggestion was
> > > >> >> > > > to
> > > >> >> > > > > add this
> > > >> >> > > > > > > > > > feature starting from the IPC(I moved initial
> > write up steps to
> > > >> >> > > > > the
> > > >> >> > > > > > > > > bottom
> > > >> >> > > > > > > > > > of the doc). Afterwards David suggested focusing
> > on Flight and
> > > >> >> > > > > that's how
> > > >> >> > > > > > > > > > we ended up with the protobufs change in the
> > proposal. This
> > > >> >> > > > > being said I
> > > >> >> > > > > > > > > do
> > > >> >> > > > > > > > > > think that the place where this should be
> > impemented is a good
> > > >> >> > > > > question
> > > >> >> > > > > > > > > on
> > > >> >> > > > > > > > > > its own. Maybe it makes sense to have this kind
> > of a feature in
> > > >> >> > > > > IPC and
> > > >> >> > > > > > > > > > somehow use it in Flight, maybe not.
> > > >> >> > > > > > > > > > 2. The point about dictionaries deserves a
> > dedicated section in
> > > >> >> > > > > the
> > > >> >> > > > > > > > > > proposal. Nate and David brought it up and
> > shared some
> > > >> >> > > > insights.
> > > >> >> > > > > I'll try
> > > >> >> > > > > > > > > > to aggregate them and we can continue the
> > discussion form
> > > >> >> > > > there.
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > Cheers,
> > > >> >> > > > > > > > > > Gosh
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > >> >> > > > > > > > > natebauernfeind@deephaven.io>
> > > >> >> > > > > > > > > > wrote:
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > > > > >
> > > >> >> > > > > > > > > > > > > > makes it more difficult to bring schema
> > evolution back
> > > >> >> > > > > into the
> > > >> >> > > > > > > > > > > > > > IPC Stream format (i.e. it would live
> > only in flight)
> > > >> >> > > > > > > > > > > > >
> > > >> >> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer
> > structures not the
> > > >> >> > > > > > > > > protobufs.
> > > >> >> > > > > > > > > > > Can
> > > >> >> > > > > > > > > > > > > you help me understand how difficult it
> > would be to bring
> > > >> >> > > > > the
> > > >> >> > > > > > > > > > > `schema_id`
> > > >> >> > > > > > > > > > > > > approach to the IPC stream format?
> > > >> >> > > > > > > > > > > >
> > > >> >> > > > > > > > > > > > I thought we were talking solely about the
> > Flight Protobuf
> > > >> >> > > > > > > > > definitions -
> > > >> >> > > > > > > > > > > > not the Flatbuffers (and the Google doc at
> > least only talks
> > > >> >> > > > > about the
> > > >> >> > > > > > > > > > > > Protobufs).
> > > >> >> > > > > > > > > > > >
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > > I somehow missed that schema_id is being added
> > to protobuf in
> > > >> >> > > > > the
> > > >> >> > > > > > > > > document.
> > > >> >> > > > > > > > > > > It feels to me that the schema_id is a
> > property that would
> > > >> >> > > > > ideally only
> > > >> >> > > > > > > > > > > apply to the RecordBatch. I better understand
> > Micah's
> > > >> >> > > > > dictionary
> > > >> >> > > > > > > > > concerns,
> > > >> >> > > > > > > > > > > now, too.
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > > > Side Question: Why isn't the IPC stream
> > format a series of
> > > >> >> > > > > the flight
> > > >> >> > > > > > > > > > > > > protobufs? It's a real shame that there is
> > no standard
> > > >> >> > > > way
> > > >> >> > > > > to
> > > >> >> > > > > > > > > > > > > capture/replay a stream with app_metadata.
> > (Obviously
> > > >> >> > > > > ignoring the
> > > >> >> > > > > > > > > > > > > annoyances around protobuf wrapping
> > flatbuffers.)
> > > >> >> > > > > > > > > > > >
> > > >> >> > > > > > > > > > > > The IPC format was defined long before
> > Flight, and Flight's
> > > >> >> > > > > > > > > app_metadata
> > > >> >> > > > > > > > > > > > was added after Flight's initial definition.
> > Note an IPC
> > > >> >> > > > > message does
> > > >> >> > > > > > > > > > > have
> > > >> >> > > > > > > > > > > > a provision for key-value metadata, though I
> > think APIs for
> > > >> >> > > > > that are
> > > >> >> > > > > > > > > not
> > > >> >> > > > > > > > > > > > fully exposed. (See ARROW-6940:
> > > >> >> > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > >> >> > > > > despite my
> > > >> >> > > > > > > > > comments
> > > >> >> > > > > > > > > > > > there perhaps we need to unify or at least
> > consider how
> > > >> >> > > > > Flight's
> > > >> >> > > > > > > > > > > > app_metadata relates to the IPC message
> > custom_metadata.
> > > >> >> > > > Also
> > > >> >> > > > > > > > > perhaps see
> > > >> >> > > > > > > > > > > > ARROW-1059.)
> > > >> >> > > > > > > > > > > >
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > > KeyValue unfortunately is string to string. In
> > flatbuffer
> > > >> >> > > > > strings are
> > > >> >> > > > > > > > > only
> > > >> >> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the
> > other hand is
> > > >> >> > > > > opaque
> > > >> >> > > > > > > > > bytes.
> > > >> >> > > > > > > > > > > The latter is a bit more useful.
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > > > --
> > > >> >> > > > > > > > > > >
> > > >> >> > > > > > > > > >
> > > >> >> > > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > > --
> > > >> >> > > >
> >
> 

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

I agree with you any thoughts on a way forward for at least hardening the
spec (or should this be done at the same time as adding the new field)?

On Mon, Aug 16, 2021 at 1:45 AM Wes McKinney <we...@gmail.com> wrote:

> I've been poking around the project, and I'm growing concerned that
> our use of the KeyValue field has already been non-compliant in many
> cases since we do not validate UTF8-ness. Since we also use KeyValue
> to handle opaque data serialization for extension types [1], the fact
> that the specification does not clarify that binary data (such as the
> output of ExtensionType::Serialize) must be base64-encoded or similar
> makes this a bit of a minefield at the moment.
>
> It seems that there are no particularly excellent solutions, and
> maintaining the status quo (having now identified these
> inconsistencies / vagueries in at least the C++ implementation) is
> probably not a good idea either. In Parquet, where we store a
> serialized binary Arrow schema, we have to base64-encode [2]
>
> [1]:
> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
> [2]:
> https://github.com/apache/arrow/blob/e990d177b1f1dec962315487682f613d46be573c/cpp/src/parquet/arrow/writer.cc#L423
>
> On Tue, Aug 10, 2021 at 3:27 PM Wes McKinney <we...@gmail.com> wrote:
> >
> > Ah, that's definitely a no-go then (I believe we verify messages
> > unconditionally in C++). That's unfortunate (and I feel responsible
> > for missing this, but I suppose we had a lot of opportunities to fix
> > it prior to the 1.0.0 format version) — so to have actual binary
> > values (which was the intention in the first place for the metadata)
> > we would need to add a new metadata field.
> >
> > On Tue, Aug 10, 2021 at 6:53 AM Micah Kornfield <em...@gmail.com>
> wrote:
> > >
> > > One issue with changing it to byte is it would effectively break any
> reader that is validating flatbuffer data, because flatbuffers verifies
> null termination [1].  While this might comply with forward compatibility
> guarantees it seems like a pretty large blast radius.
> > >
> > > [1]
> https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flatbuffers.h#L2457
> > >
> > > On Mon, Jul 12, 2021 at 12:38 PM Wes McKinney <we...@gmail.com>
> wrote:
> > >>
> > >> pyarrow at least treats the KeyValue values as binary and not UTF-8.
> > >>
> > >> On Sun, Jul 11, 2021 at 9:40 PM Micah Kornfield <
> emkornfield@gmail.com> wrote:
> > >> >
> > >> > I think other languages (e.g. java, python) might make more of
> distinction between utf-8 compatible strings and raw bytes.  For python it
> might be less of a concern if the c++ wrapper already makes the value field
> look like a bytes field
> > >> >
> > >> > On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:
> > >> >>
> > >> >> We could certainly "upgrade" KeyValue to have a binary value field
> > >> >> everywhere KeyValue is used, but there is some risk of code in the
> > >> >> wild expecting there to be a null terminator after the string data.
> > >> >> The Flatbuffers-generated accessor APIs do not depend on the
> existence
> > >> >> of the null terminator, though. Not ideal, but I would not be
> thrilled
> > >> >> about adding an extra [ BinaryKeyValue ] everyplace we currently
> have
> > >> >> [ KeyValue ].
> > >> >>
> > >> >> That said, I doubt that we have any endogenous forward
> compatibility
> > >> >> problems related to this in Apache Arrow-maintained libraries, the
> > >> >> risk would come from users who are interacting with the Flatbuffers
> > >> >> data manually / without using one of our libraries. We could
> implement
> > >> >> the changes and run a set of forward compatibility integration
> tests
> > >> >> to see if anyone of our released libraries have an issue.
> > >> >>
> > >> >> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com>
> wrote:
> > >> >> >
> > >> >> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
> > >> >> >
> > >> >> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <
> emkornfield@gmail.com> wrote:
> > >> >> > >
> > >> >> > > Retitling and forking the discussion to talk about key value
> pairs.
> > >> >> > >
> > >> >> > > What is the byte cost of an empty list?  Another option would
> be to
> > >> >> > > introduce a new BinaryKeyValue table and add binary metadata.
> > >> >> > >
> > >> >> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > >> >> > > natebauernfeind@deephaven.io> wrote:
> > >> >> > >
> > >> >> > > > Deephaven and I are very supportive of "upgrading" the value
> half of the kv
> > >> >> > > > pair to a byte vector. What is the best way to find out if
> there is
> > >> >> > > > sufficient interest?
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > I've been stewing on the ideas here around schema evolution,
> and I realize
> > >> >> > > > the specific feature I am missing is the ability to encode
> that a field
> > >> >> > > > (i.e. its FieldNode and accompanying Buffers in the
> RecordBatch) is
> > >> >> > > > empty/has-no-data in O(0) cost (yes; for free).
> > >> >> > > >
> > >> >> > > > Might there be interest in adding a "field_id" to the
> FieldNode (which is
> > >> >> > > > encoded on the RecordBatch flatbuffer)? I see a simple
> forward-compatible
> > >> >> > > > upgrade (by either keying off of 0, or explicitly set the
> field default to
> > >> >> > > > -1) which would allow the sender to "skip" fields that have
> 1) FieldNode
> > >> >> > > > length of zero, and 2) all Buffer's associated at that level
> (and further
> > >> >> > > > nested) are also equally empty (i.e. Buffer length is zero).
> > >> >> > > >
> > >> >> > > > I understand this concept slightly interferes with
> RecordBatch's `length`
> > >> >> > > > field, and that many implementations use that length to
> resize the
> > >> >> > > > root-level FieldNodes. The use-case I have in mind has
> different logical
> > >> >> > > > lengths per field node; current implementations require
> sending a
> > >> >> > > > RecordBatch length of the max length across all root level
> field nodes. I
> > >> >> > > > believe this requires a copy of data whenever a field node
> is too short; I
> > >> >> > > > don't know if there is a decent solution to this slight
> inefficiency. I am
> > >> >> > > > bringing it up because if "skipping a field node when it is
> empty" is a
> > >> >> > > > feature, then we may not want to allocate space for those
> nodes given that
> > >> >> > > > the record batch length will likely be greater than zero.
> > >> >> > > >
> > >> >> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <
> wesmckinn@gmail.com> wrote:
> > >> >> > > >
> > >> >> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <
> apache@lidavidm.me> wrote:
> > >> >> > > > > >
> > >> >> > > > > > From the Flatbuffers internals doc[1] it appears they
> are the same:
> > >> >> > > > > "Strings are simply a vector of bytes, and are always
> null-terminated."
> > >> >> > > > >
> > >> >> > > > > I see. I took a look at flatbuffers.h, and it appears that
> changing
> > >> >> > > > > this field from string to [byte] would be
> backward-compatible and
> > >> >> > > > > forward-compatible except with code that expects a null
> terminator.
> > >> >> > > > > This is something we could discuss separately if there
> were enough
> > >> >> > > > > interest.
> > >> >> > > > >
> > >> >> > > > > > [1]:
> https://google.github.io/flatbuffers/flatbuffers_internals.html
> > >> >> > > > > >
> > >> >> > > > > > -David
> > >> >> > > > > >
> > >> >> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > >> >> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > >> >> > > > emkornfield@gmail.com>
> > >> >> > > > > wrote:
> > >> >> > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Right, I had wanted to focus the discussion on
> Flight as I think
> > >> >> > > > > schema
> > >> >> > > > > > > > > evolution or multiplexing streams (more so the
> latter) is a
> > >> >> > > > > property of the
> > >> >> > > > > > > > > transport and not the stream format itself. If we
> are leaning
> > >> >> > > > > towards just
> > >> >> > > > > > > > > schema evolution then maybe it makes sense to
> discuss it for the
> > >> >> > > > > IPC stream
> > >> >> > > > > > > > > format and leverage that in Flight. I'd be
> interested in what
> > >> >> > > > > others think.
> > >> >> > > > > > > >
> > >> >> > > > > > > > I tend to agree, I think stream multiplexing is
> likely a transport
> > >> >> > > > > level
> > >> >> > > > > > > > issue.  IMO I think schema evolution should be
> consistent with the
> > >> >> > > > > IPC
> > >> >> > > > > > > > stream format  and flight.
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > > Nate: it may be worth starting a separate
> discussion about more
> > >> >> > > > > general
> > >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> key-value
> > >> >> > > > > metadata was
> > >> >> > > > > > > > > chosen/if opaque bytes were considered in the past.
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > I think  this was an unfortunate design of the key
> value metadata
> > >> >> > > > in
> > >> >> > > > > > > > Schema.fbs, but I don't think I was around when this
> decision was
> > >> >> > > > > made.
> > >> >> > > > > > >
> > >> >> > > > > > > I agree that it's unfortunate that we did not use [
> byte ] instead of
> > >> >> > > > > > > string for the value in the KeyValue metadata — I
> think this was more
> > >> >> > > > > > > of an oversight than a deliberate choice (e.g. it was
> not our intent
> > >> >> > > > > > > to require binary data to be base64-encoded — this is
> something that
> > >> >> > > > > > > we have to do when encoding binary data in Thrift
> KeyValue metadata
> > >> >> > > > > > > for Parquet, for example). Is the binary
> representation of [byte]
> > >> >> > > > > > > different from string?
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > >
> > >> >> > > > > > > > Side Question: Why isn't the IPC stream format a
> series of the
> > >> >> > > > flight
> > >> >> > > > > > > > > protobufs?
> > >> >> > > > > > > >
> > >> >> > > > > > > > In addition to what David said, protobufs can't be
> read directly
> > >> >> > > > > from a
> > >> >> > > > > > > > memory-mapped file (they need decoding).  This was
> one of the
> > >> >> > > > design
> > >> >> > > > > > > > considerations of using flatbuffers and IPC
> Stream/File format.
> > >> >> > > > > > > >
> > >> >> > > > > > > > I was thinking Micah's comment is more that whatever
> we do, it
> > >> >> > > > > should be
> > >> >> > > > > > > > > clearly specified and edge cases should be
> considered, especially
> > >> >> > > > > if we
> > >> >> > > > > > > > > might want to 'backport' this into the stream
> format later.
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > Yes, for dictionaries we just need to be careful to
> define
> > >> >> > > > semantics
> > >> >> > > > > and
> > >> >> > > > > > > > ensure implementations are validating them with
> regards to
> > >> >> > > > > dictionaries.
> > >> >> > > > > > > > There likely isn't any need to change current
> implementations
> > >> >> > > > though.
> > >> >> > > > > > > >
> > >> >> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <
> lidavidm@apache.org>
> > >> >> > > > > wrote:
> > >> >> > > > > > > >
> > >> >> > > > > > > > > Right, I had wanted to focus the discussion on
> Flight as I think
> > >> >> > > > > schema
> > >> >> > > > > > > > > evolution or multiplexing streams (more so the
> latter) is a
> > >> >> > > > > property of the
> > >> >> > > > > > > > > transport and not the stream format itself. If we
> are leaning
> > >> >> > > > > towards just
> > >> >> > > > > > > > > schema evolution then maybe it makes sense to
> discuss it for the
> > >> >> > > > > IPC stream
> > >> >> > > > > > > > > format and leverage that in Flight. I'd be
> interested in what
> > >> >> > > > > others think.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Especially if we are looking at multiplexing
> streams - I would
> > >> >> > > > > wonder if
> > >> >> > > > > > > > > that's actually better served by making it easier
> to implement
> > >> >> > > > > using the
> > >> >> > > > > > > > > Flight implementation as it stands (by managing
> concurrent RPC
> > >> >> > > > > calls and/or
> > >> >> > > > > > > > > performing the union-of-structs encoding trick for
> you), instead
> > >> >> > > > > of having
> > >> >> > > > > > > > > to change the protocol.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > Nate: it may be worth starting a separate
> discussion about more
> > >> >> > > > > general
> > >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> key-value
> > >> >> > > > > metadata was
> > >> >> > > > > > > > > chosen/if opaque bytes were considered in the
> past. Off the top
> > >> >> > > > of
> > >> >> > > > > my head
> > >> >> > > > > > > > > if it's for on-disk storage and fully
> application-defined it may
> > >> >> > > > > make sense
> > >> >> > > > > > > > > to store as a separate file alongside the Arrow
> file (indexed by
> > >> >> > > > > record
> > >> >> > > > > > > > > batch index) where you can take advantage of
> whatever format is
> > >> >> > > > > most
> > >> >> > > > > > > > > suitable.
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > -David
> > >> >> > > > > > > > >
> > >> >> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan
> wrote:
> > >> >> > > > > > > > > > Hi guys,
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial
> suggestion was
> > >> >> > > > to
> > >> >> > > > > add this
> > >> >> > > > > > > > > > feature starting from the IPC(I moved initial
> write up steps to
> > >> >> > > > > the
> > >> >> > > > > > > > > bottom
> > >> >> > > > > > > > > > of the doc). Afterwards David suggested focusing
> on Flight and
> > >> >> > > > > that's how
> > >> >> > > > > > > > > > we ended up with the protobufs change in the
> proposal. This
> > >> >> > > > > being said I
> > >> >> > > > > > > > > do
> > >> >> > > > > > > > > > think that the place where this should be
> impemented is a good
> > >> >> > > > > question
> > >> >> > > > > > > > > on
> > >> >> > > > > > > > > > its own. Maybe it makes sense to have this kind
> of a feature in
> > >> >> > > > > IPC and
> > >> >> > > > > > > > > > somehow use it in Flight, maybe not.
> > >> >> > > > > > > > > > 2. The point about dictionaries deserves a
> dedicated section in
> > >> >> > > > > the
> > >> >> > > > > > > > > > proposal. Nate and David brought it up and
> shared some
> > >> >> > > > insights.
> > >> >> > > > > I'll try
> > >> >> > > > > > > > > > to aggregate them and we can continue the
> discussion form
> > >> >> > > > there.
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > Cheers,
> > >> >> > > > > > > > > > Gosh
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > >> >> > > > > > > > > natebauernfeind@deephaven.io>
> > >> >> > > > > > > > > > wrote:
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > > > > >
> > >> >> > > > > > > > > > > > > > makes it more difficult to bring schema
> evolution back
> > >> >> > > > > into the
> > >> >> > > > > > > > > > > > > > IPC Stream format (i.e. it would live
> only in flight)
> > >> >> > > > > > > > > > > > >
> > >> >> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer
> structures not the
> > >> >> > > > > > > > > protobufs.
> > >> >> > > > > > > > > > > Can
> > >> >> > > > > > > > > > > > > you help me understand how difficult it
> would be to bring
> > >> >> > > > > the
> > >> >> > > > > > > > > > > `schema_id`
> > >> >> > > > > > > > > > > > > approach to the IPC stream format?
> > >> >> > > > > > > > > > > >
> > >> >> > > > > > > > > > > > I thought we were talking solely about the
> Flight Protobuf
> > >> >> > > > > > > > > definitions -
> > >> >> > > > > > > > > > > > not the Flatbuffers (and the Google doc at
> least only talks
> > >> >> > > > > about the
> > >> >> > > > > > > > > > > > Protobufs).
> > >> >> > > > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > > I somehow missed that schema_id is being added
> to protobuf in
> > >> >> > > > > the
> > >> >> > > > > > > > > document.
> > >> >> > > > > > > > > > > It feels to me that the schema_id is a
> property that would
> > >> >> > > > > ideally only
> > >> >> > > > > > > > > > > apply to the RecordBatch. I better understand
> Micah's
> > >> >> > > > > dictionary
> > >> >> > > > > > > > > concerns,
> > >> >> > > > > > > > > > > now, too.
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > > > Side Question: Why isn't the IPC stream
> format a series of
> > >> >> > > > > the flight
> > >> >> > > > > > > > > > > > > protobufs? It's a real shame that there is
> no standard
> > >> >> > > > way
> > >> >> > > > > to
> > >> >> > > > > > > > > > > > > capture/replay a stream with app_metadata.
> (Obviously
> > >> >> > > > > ignoring the
> > >> >> > > > > > > > > > > > > annoyances around protobuf wrapping
> flatbuffers.)
> > >> >> > > > > > > > > > > >
> > >> >> > > > > > > > > > > > The IPC format was defined long before
> Flight, and Flight's
> > >> >> > > > > > > > > app_metadata
> > >> >> > > > > > > > > > > > was added after Flight's initial definition.
> Note an IPC
> > >> >> > > > > message does
> > >> >> > > > > > > > > > > have
> > >> >> > > > > > > > > > > > a provision for key-value metadata, though I
> think APIs for
> > >> >> > > > > that are
> > >> >> > > > > > > > > not
> > >> >> > > > > > > > > > > > fully exposed. (See ARROW-6940:
> > >> >> > > > > > > > > > > >
> https://issues.apache.org/jira/browse/ARROW-6940 and
> > >> >> > > > > despite my
> > >> >> > > > > > > > > comments
> > >> >> > > > > > > > > > > > there perhaps we need to unify or at least
> consider how
> > >> >> > > > > Flight's
> > >> >> > > > > > > > > > > > app_metadata relates to the IPC message
> custom_metadata.
> > >> >> > > > Also
> > >> >> > > > > > > > > perhaps see
> > >> >> > > > > > > > > > > > ARROW-1059.)
> > >> >> > > > > > > > > > > >
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > > KeyValue unfortunately is string to string. In
> flatbuffer
> > >> >> > > > > strings are
> > >> >> > > > > > > > > only
> > >> >> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the
> other hand is
> > >> >> > > > > opaque
> > >> >> > > > > > > > > bytes.
> > >> >> > > > > > > > > > > The latter is a bit more useful.
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > > > --
> > >> >> > > > > > > > > > >
> > >> >> > > > > > > > > >
> > >> >> > > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > > >
> > >> >> > > > --
> > >> >> > > >
>

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

I've been poking around the project, and I'm growing concerned that
our use of the KeyValue field has already been non-compliant in many
cases since we do not validate UTF8-ness. Since we also use KeyValue
to handle opaque data serialization for extension types [1], the fact
that the specification does not clarify that binary data (such as the
output of ExtensionType::Serialize) must be base64-encoded or similar
makes this a bit of a minefield at the moment.

It seems that there are no particularly excellent solutions, and
maintaining the status quo (having now identified these
inconsistencies / vagueries in at least the C++ implementation) is
probably not a good idea either. In Parquet, where we store a
serialized binary Arrow schema, we have to base64-encode [2]

[1]: https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
[2]: https://github.com/apache/arrow/blob/e990d177b1f1dec962315487682f613d46be573c/cpp/src/parquet/arrow/writer.cc#L423

On Tue, Aug 10, 2021 at 3:27 PM Wes McKinney <we...@gmail.com> wrote:
>
> Ah, that's definitely a no-go then (I believe we verify messages
> unconditionally in C++). That's unfortunate (and I feel responsible
> for missing this, but I suppose we had a lot of opportunities to fix
> it prior to the 1.0.0 format version) — so to have actual binary
> values (which was the intention in the first place for the metadata)
> we would need to add a new metadata field.
>
> On Tue, Aug 10, 2021 at 6:53 AM Micah Kornfield <em...@gmail.com> wrote:
> >
> > One issue with changing it to byte is it would effectively break any reader that is validating flatbuffer data, because flatbuffers verifies null termination [1].  While this might comply with forward compatibility guarantees it seems like a pretty large blast radius.
> >
> > [1] https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flatbuffers.h#L2457
> >
> > On Mon, Jul 12, 2021 at 12:38 PM Wes McKinney <we...@gmail.com> wrote:
> >>
> >> pyarrow at least treats the KeyValue values as binary and not UTF-8.
> >>
> >> On Sun, Jul 11, 2021 at 9:40 PM Micah Kornfield <em...@gmail.com> wrote:
> >> >
> >> > I think other languages (e.g. java, python) might make more of distinction between utf-8 compatible strings and raw bytes.  For python it might be less of a concern if the c++ wrapper already makes the value field look like a bytes field
> >> >
> >> > On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:
> >> >>
> >> >> We could certainly "upgrade" KeyValue to have a binary value field
> >> >> everywhere KeyValue is used, but there is some risk of code in the
> >> >> wild expecting there to be a null terminator after the string data.
> >> >> The Flatbuffers-generated accessor APIs do not depend on the existence
> >> >> of the null terminator, though. Not ideal, but I would not be thrilled
> >> >> about adding an extra [ BinaryKeyValue ] everyplace we currently have
> >> >> [ KeyValue ].
> >> >>
> >> >> That said, I doubt that we have any endogenous forward compatibility
> >> >> problems related to this in Apache Arrow-maintained libraries, the
> >> >> risk would come from users who are interacting with the Flatbuffers
> >> >> data manually / without using one of our libraries. We could implement
> >> >> the changes and run a set of forward compatibility integration tests
> >> >> to see if anyone of our released libraries have an issue.
> >> >>
> >> >> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com> wrote:
> >> >> >
> >> >> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
> >> >> >
> >> >> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <em...@gmail.com> wrote:
> >> >> > >
> >> >> > > Retitling and forking the discussion to talk about key value pairs.
> >> >> > >
> >> >> > > What is the byte cost of an empty list?  Another option would be to
> >> >> > > introduce a new BinaryKeyValue table and add binary metadata.
> >> >> > >
> >> >> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> >> >> > > natebauernfeind@deephaven.io> wrote:
> >> >> > >
> >> >> > > > Deephaven and I are very supportive of "upgrading" the value half of the kv
> >> >> > > > pair to a byte vector. What is the best way to find out if there is
> >> >> > > > sufficient interest?
> >> >> > > >
> >> >> > > >
> >> >> > > > I've been stewing on the ideas here around schema evolution, and I realize
> >> >> > > > the specific feature I am missing is the ability to encode that a field
> >> >> > > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> >> >> > > > empty/has-no-data in O(0) cost (yes; for free).
> >> >> > > >
> >> >> > > > Might there be interest in adding a "field_id" to the FieldNode (which is
> >> >> > > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> >> >> > > > upgrade (by either keying off of 0, or explicitly set the field default to
> >> >> > > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> >> >> > > > length of zero, and 2) all Buffer's associated at that level (and further
> >> >> > > > nested) are also equally empty (i.e. Buffer length is zero).
> >> >> > > >
> >> >> > > > I understand this concept slightly interferes with RecordBatch's `length`
> >> >> > > > field, and that many implementations use that length to resize the
> >> >> > > > root-level FieldNodes. The use-case I have in mind has different logical
> >> >> > > > lengths per field node; current implementations require sending a
> >> >> > > > RecordBatch length of the max length across all root level field nodes. I
> >> >> > > > believe this requires a copy of data whenever a field node is too short; I
> >> >> > > > don't know if there is a decent solution to this slight inefficiency. I am
> >> >> > > > bringing it up because if "skipping a field node when it is empty" is a
> >> >> > > > feature, then we may not want to allocate space for those nodes given that
> >> >> > > > the record batch length will likely be greater than zero.
> >> >> > > >
> >> >> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
> >> >> > > >
> >> >> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> >> >> > > > > >
> >> >> > > > > > From the Flatbuffers internals doc[1] it appears they are the same:
> >> >> > > > > "Strings are simply a vector of bytes, and are always null-terminated."
> >> >> > > > >
> >> >> > > > > I see. I took a look at flatbuffers.h, and it appears that changing
> >> >> > > > > this field from string to [byte] would be backward-compatible and
> >> >> > > > > forward-compatible except with code that expects a null terminator.
> >> >> > > > > This is something we could discuss separately if there were enough
> >> >> > > > > interest.
> >> >> > > > >
> >> >> > > > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> >> >> > > > > >
> >> >> > > > > > -David
> >> >> > > > > >
> >> >> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> >> >> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> >> >> > > > emkornfield@gmail.com>
> >> >> > > > > wrote:
> >> >> > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
> >> >> > > > > schema
> >> >> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
> >> >> > > > > property of the
> >> >> > > > > > > > > transport and not the stream format itself. If we are leaning
> >> >> > > > > towards just
> >> >> > > > > > > > > schema evolution then maybe it makes sense to discuss it for the
> >> >> > > > > IPC stream
> >> >> > > > > > > > > format and leverage that in Flight. I'd be interested in what
> >> >> > > > > others think.
> >> >> > > > > > > >
> >> >> > > > > > > > I tend to agree, I think stream multiplexing is likely a transport
> >> >> > > > > level
> >> >> > > > > > > > issue.  IMO I think schema evolution should be consistent with the
> >> >> > > > > IPC
> >> >> > > > > > > > stream format  and flight.
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > > Nate: it may be worth starting a separate discussion about more
> >> >> > > > > general
> >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> >> >> > > > > metadata was
> >> >> > > > > > > > > chosen/if opaque bytes were considered in the past.
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > I think  this was an unfortunate design of the key value metadata
> >> >> > > > in
> >> >> > > > > > > > Schema.fbs, but I don't think I was around when this decision was
> >> >> > > > > made.
> >> >> > > > > > >
> >> >> > > > > > > I agree that it's unfortunate that we did not use [ byte ] instead of
> >> >> > > > > > > string for the value in the KeyValue metadata — I think this was more
> >> >> > > > > > > of an oversight than a deliberate choice (e.g. it was not our intent
> >> >> > > > > > > to require binary data to be base64-encoded — this is something that
> >> >> > > > > > > we have to do when encoding binary data in Thrift KeyValue metadata
> >> >> > > > > > > for Parquet, for example). Is the binary representation of [byte]
> >> >> > > > > > > different from string?
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > >
> >> >> > > > > > > > Side Question: Why isn't the IPC stream format a series of the
> >> >> > > > flight
> >> >> > > > > > > > > protobufs?
> >> >> > > > > > > >
> >> >> > > > > > > > In addition to what David said, protobufs can't be read directly
> >> >> > > > > from a
> >> >> > > > > > > > memory-mapped file (they need decoding).  This was one of the
> >> >> > > > design
> >> >> > > > > > > > considerations of using flatbuffers and IPC Stream/File format.
> >> >> > > > > > > >
> >> >> > > > > > > > I was thinking Micah's comment is more that whatever we do, it
> >> >> > > > > should be
> >> >> > > > > > > > > clearly specified and edge cases should be considered, especially
> >> >> > > > > if we
> >> >> > > > > > > > > might want to 'backport' this into the stream format later.
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > Yes, for dictionaries we just need to be careful to define
> >> >> > > > semantics
> >> >> > > > > and
> >> >> > > > > > > > ensure implementations are validating them with regards to
> >> >> > > > > dictionaries.
> >> >> > > > > > > > There likely isn't any need to change current implementations
> >> >> > > > though.
> >> >> > > > > > > >
> >> >> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> >> >> > > > > wrote:
> >> >> > > > > > > >
> >> >> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
> >> >> > > > > schema
> >> >> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
> >> >> > > > > property of the
> >> >> > > > > > > > > transport and not the stream format itself. If we are leaning
> >> >> > > > > towards just
> >> >> > > > > > > > > schema evolution then maybe it makes sense to discuss it for the
> >> >> > > > > IPC stream
> >> >> > > > > > > > > format and leverage that in Flight. I'd be interested in what
> >> >> > > > > others think.
> >> >> > > > > > > > >
> >> >> > > > > > > > > Especially if we are looking at multiplexing streams - I would
> >> >> > > > > wonder if
> >> >> > > > > > > > > that's actually better served by making it easier to implement
> >> >> > > > > using the
> >> >> > > > > > > > > Flight implementation as it stands (by managing concurrent RPC
> >> >> > > > > calls and/or
> >> >> > > > > > > > > performing the union-of-structs encoding trick for you), instead
> >> >> > > > > of having
> >> >> > > > > > > > > to change the protocol.
> >> >> > > > > > > > >
> >> >> > > > > > > > > Nate: it may be worth starting a separate discussion about more
> >> >> > > > > general
> >> >> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> >> >> > > > > metadata was
> >> >> > > > > > > > > chosen/if opaque bytes were considered in the past. Off the top
> >> >> > > > of
> >> >> > > > > my head
> >> >> > > > > > > > > if it's for on-disk storage and fully application-defined it may
> >> >> > > > > make sense
> >> >> > > > > > > > > to store as a separate file alongside the Arrow file (indexed by
> >> >> > > > > record
> >> >> > > > > > > > > batch index) where you can take advantage of whatever format is
> >> >> > > > > most
> >> >> > > > > > > > > suitable.
> >> >> > > > > > > > >
> >> >> > > > > > > > > -David
> >> >> > > > > > > > >
> >> >> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> >> >> > > > > > > > > > Hi guys,
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> >> >> > > > to
> >> >> > > > > add this
> >> >> > > > > > > > > > feature starting from the IPC(I moved initial write up steps to
> >> >> > > > > the
> >> >> > > > > > > > > bottom
> >> >> > > > > > > > > > of the doc). Afterwards David suggested focusing on Flight and
> >> >> > > > > that's how
> >> >> > > > > > > > > > we ended up with the protobufs change in the proposal. This
> >> >> > > > > being said I
> >> >> > > > > > > > > do
> >> >> > > > > > > > > > think that the place where this should be impemented is a good
> >> >> > > > > question
> >> >> > > > > > > > > on
> >> >> > > > > > > > > > its own. Maybe it makes sense to have this kind of a feature in
> >> >> > > > > IPC and
> >> >> > > > > > > > > > somehow use it in Flight, maybe not.
> >> >> > > > > > > > > > 2. The point about dictionaries deserves a dedicated section in
> >> >> > > > > the
> >> >> > > > > > > > > > proposal. Nate and David brought it up and shared some
> >> >> > > > insights.
> >> >> > > > > I'll try
> >> >> > > > > > > > > > to aggregate them and we can continue the discussion form
> >> >> > > > there.
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > Cheers,
> >> >> > > > > > > > > > Gosh
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> >> >> > > > > > > > > natebauernfeind@deephaven.io>
> >> >> > > > > > > > > > wrote:
> >> >> > > > > > > > > >
> >> >> > > > > > > > > > > >
> >> >> > > > > > > > > > > > > > makes it more difficult to bring schema evolution back
> >> >> > > > > into the
> >> >> > > > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> >> >> > > > > > > > > > > > >
> >> >> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
> >> >> > > > > > > > > protobufs.
> >> >> > > > > > > > > > > Can
> >> >> > > > > > > > > > > > > you help me understand how difficult it would be to bring
> >> >> > > > > the
> >> >> > > > > > > > > > > `schema_id`
> >> >> > > > > > > > > > > > > approach to the IPC stream format?
> >> >> > > > > > > > > > > >
> >> >> > > > > > > > > > > > I thought we were talking solely about the Flight Protobuf
> >> >> > > > > > > > > definitions -
> >> >> > > > > > > > > > > > not the Flatbuffers (and the Google doc at least only talks
> >> >> > > > > about the
> >> >> > > > > > > > > > > > Protobufs).
> >> >> > > > > > > > > > > >
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > > I somehow missed that schema_id is being added to protobuf in
> >> >> > > > > the
> >> >> > > > > > > > > document.
> >> >> > > > > > > > > > > It feels to me that the schema_id is a property that would
> >> >> > > > > ideally only
> >> >> > > > > > > > > > > apply to the RecordBatch. I better understand Micah's
> >> >> > > > > dictionary
> >> >> > > > > > > > > concerns,
> >> >> > > > > > > > > > > now, too.
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > > > Side Question: Why isn't the IPC stream format a series of
> >> >> > > > > the flight
> >> >> > > > > > > > > > > > > protobufs? It's a real shame that there is no standard
> >> >> > > > way
> >> >> > > > > to
> >> >> > > > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> >> >> > > > > ignoring the
> >> >> > > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> >> >> > > > > > > > > > > >
> >> >> > > > > > > > > > > > The IPC format was defined long before Flight, and Flight's
> >> >> > > > > > > > > app_metadata
> >> >> > > > > > > > > > > > was added after Flight's initial definition. Note an IPC
> >> >> > > > > message does
> >> >> > > > > > > > > > > have
> >> >> > > > > > > > > > > > a provision for key-value metadata, though I think APIs for
> >> >> > > > > that are
> >> >> > > > > > > > > not
> >> >> > > > > > > > > > > > fully exposed. (See ARROW-6940:
> >> >> > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> >> >> > > > > despite my
> >> >> > > > > > > > > comments
> >> >> > > > > > > > > > > > there perhaps we need to unify or at least consider how
> >> >> > > > > Flight's
> >> >> > > > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> >> >> > > > Also
> >> >> > > > > > > > > perhaps see
> >> >> > > > > > > > > > > > ARROW-1059.)
> >> >> > > > > > > > > > > >
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> >> >> > > > > strings are
> >> >> > > > > > > > > only
> >> >> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> >> >> > > > > opaque
> >> >> > > > > > > > > bytes.
> >> >> > > > > > > > > > > The latter is a bit more useful.
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > > > --
> >> >> > > > > > > > > > >
> >> >> > > > > > > > > >
> >> >> > > > > > > > >
> >> >> > > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > > >
> >> >> > > > --
> >> >> > > >

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

Ah, that's definitely a no-go then (I believe we verify messages
unconditionally in C++). That's unfortunate (and I feel responsible
for missing this, but I suppose we had a lot of opportunities to fix
it prior to the 1.0.0 format version) — so to have actual binary
values (which was the intention in the first place for the metadata)
we would need to add a new metadata field.

On Tue, Aug 10, 2021 at 6:53 AM Micah Kornfield <em...@gmail.com> wrote:
>
> One issue with changing it to byte is it would effectively break any reader that is validating flatbuffer data, because flatbuffers verifies null termination [1].  While this might comply with forward compatibility guarantees it seems like a pretty large blast radius.
>
> [1] https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flatbuffers.h#L2457
>
> On Mon, Jul 12, 2021 at 12:38 PM Wes McKinney <we...@gmail.com> wrote:
>>
>> pyarrow at least treats the KeyValue values as binary and not UTF-8.
>>
>> On Sun, Jul 11, 2021 at 9:40 PM Micah Kornfield <em...@gmail.com> wrote:
>> >
>> > I think other languages (e.g. java, python) might make more of distinction between utf-8 compatible strings and raw bytes.  For python it might be less of a concern if the c++ wrapper already makes the value field look like a bytes field
>> >
>> > On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:
>> >>
>> >> We could certainly "upgrade" KeyValue to have a binary value field
>> >> everywhere KeyValue is used, but there is some risk of code in the
>> >> wild expecting there to be a null terminator after the string data.
>> >> The Flatbuffers-generated accessor APIs do not depend on the existence
>> >> of the null terminator, though. Not ideal, but I would not be thrilled
>> >> about adding an extra [ BinaryKeyValue ] everyplace we currently have
>> >> [ KeyValue ].
>> >>
>> >> That said, I doubt that we have any endogenous forward compatibility
>> >> problems related to this in Apache Arrow-maintained libraries, the
>> >> risk would come from users who are interacting with the Flatbuffers
>> >> data manually / without using one of our libraries. We could implement
>> >> the changes and run a set of forward compatibility integration tests
>> >> to see if anyone of our released libraries have an issue.
>> >>
>> >> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com> wrote:
>> >> >
>> >> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
>> >> >
>> >> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <em...@gmail.com> wrote:
>> >> > >
>> >> > > Retitling and forking the discussion to talk about key value pairs.
>> >> > >
>> >> > > What is the byte cost of an empty list?  Another option would be to
>> >> > > introduce a new BinaryKeyValue table and add binary metadata.
>> >> > >
>> >> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
>> >> > > natebauernfeind@deephaven.io> wrote:
>> >> > >
>> >> > > > Deephaven and I are very supportive of "upgrading" the value half of the kv
>> >> > > > pair to a byte vector. What is the best way to find out if there is
>> >> > > > sufficient interest?
>> >> > > >
>> >> > > >
>> >> > > > I've been stewing on the ideas here around schema evolution, and I realize
>> >> > > > the specific feature I am missing is the ability to encode that a field
>> >> > > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
>> >> > > > empty/has-no-data in O(0) cost (yes; for free).
>> >> > > >
>> >> > > > Might there be interest in adding a "field_id" to the FieldNode (which is
>> >> > > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
>> >> > > > upgrade (by either keying off of 0, or explicitly set the field default to
>> >> > > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
>> >> > > > length of zero, and 2) all Buffer's associated at that level (and further
>> >> > > > nested) are also equally empty (i.e. Buffer length is zero).
>> >> > > >
>> >> > > > I understand this concept slightly interferes with RecordBatch's `length`
>> >> > > > field, and that many implementations use that length to resize the
>> >> > > > root-level FieldNodes. The use-case I have in mind has different logical
>> >> > > > lengths per field node; current implementations require sending a
>> >> > > > RecordBatch length of the max length across all root level field nodes. I
>> >> > > > believe this requires a copy of data whenever a field node is too short; I
>> >> > > > don't know if there is a decent solution to this slight inefficiency. I am
>> >> > > > bringing it up because if "skipping a field node when it is empty" is a
>> >> > > > feature, then we may not want to allocate space for those nodes given that
>> >> > > > the record batch length will likely be greater than zero.
>> >> > > >
>> >> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
>> >> > > >
>> >> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
>> >> > > > > >
>> >> > > > > > From the Flatbuffers internals doc[1] it appears they are the same:
>> >> > > > > "Strings are simply a vector of bytes, and are always null-terminated."
>> >> > > > >
>> >> > > > > I see. I took a look at flatbuffers.h, and it appears that changing
>> >> > > > > this field from string to [byte] would be backward-compatible and
>> >> > > > > forward-compatible except with code that expects a null terminator.
>> >> > > > > This is something we could discuss separately if there were enough
>> >> > > > > interest.
>> >> > > > >
>> >> > > > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
>> >> > > > > >
>> >> > > > > > -David
>> >> > > > > >
>> >> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
>> >> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
>> >> > > > emkornfield@gmail.com>
>> >> > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
>> >> > > > > schema
>> >> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
>> >> > > > > property of the
>> >> > > > > > > > > transport and not the stream format itself. If we are leaning
>> >> > > > > towards just
>> >> > > > > > > > > schema evolution then maybe it makes sense to discuss it for the
>> >> > > > > IPC stream
>> >> > > > > > > > > format and leverage that in Flight. I'd be interested in what
>> >> > > > > others think.
>> >> > > > > > > >
>> >> > > > > > > > I tend to agree, I think stream multiplexing is likely a transport
>> >> > > > > level
>> >> > > > > > > > issue.  IMO I think schema evolution should be consistent with the
>> >> > > > > IPC
>> >> > > > > > > > stream format  and flight.
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > > Nate: it may be worth starting a separate discussion about more
>> >> > > > > general
>> >> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
>> >> > > > > metadata was
>> >> > > > > > > > > chosen/if opaque bytes were considered in the past.
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > I think  this was an unfortunate design of the key value metadata
>> >> > > > in
>> >> > > > > > > > Schema.fbs, but I don't think I was around when this decision was
>> >> > > > > made.
>> >> > > > > > >
>> >> > > > > > > I agree that it's unfortunate that we did not use [ byte ] instead of
>> >> > > > > > > string for the value in the KeyValue metadata — I think this was more
>> >> > > > > > > of an oversight than a deliberate choice (e.g. it was not our intent
>> >> > > > > > > to require binary data to be base64-encoded — this is something that
>> >> > > > > > > we have to do when encoding binary data in Thrift KeyValue metadata
>> >> > > > > > > for Parquet, for example). Is the binary representation of [byte]
>> >> > > > > > > different from string?
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > > Side Question: Why isn't the IPC stream format a series of the
>> >> > > > flight
>> >> > > > > > > > > protobufs?
>> >> > > > > > > >
>> >> > > > > > > > In addition to what David said, protobufs can't be read directly
>> >> > > > > from a
>> >> > > > > > > > memory-mapped file (they need decoding).  This was one of the
>> >> > > > design
>> >> > > > > > > > considerations of using flatbuffers and IPC Stream/File format.
>> >> > > > > > > >
>> >> > > > > > > > I was thinking Micah's comment is more that whatever we do, it
>> >> > > > > should be
>> >> > > > > > > > > clearly specified and edge cases should be considered, especially
>> >> > > > > if we
>> >> > > > > > > > > might want to 'backport' this into the stream format later.
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > Yes, for dictionaries we just need to be careful to define
>> >> > > > semantics
>> >> > > > > and
>> >> > > > > > > > ensure implementations are validating them with regards to
>> >> > > > > dictionaries.
>> >> > > > > > > > There likely isn't any need to change current implementations
>> >> > > > though.
>> >> > > > > > > >
>> >> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
>> >> > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
>> >> > > > > schema
>> >> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
>> >> > > > > property of the
>> >> > > > > > > > > transport and not the stream format itself. If we are leaning
>> >> > > > > towards just
>> >> > > > > > > > > schema evolution then maybe it makes sense to discuss it for the
>> >> > > > > IPC stream
>> >> > > > > > > > > format and leverage that in Flight. I'd be interested in what
>> >> > > > > others think.
>> >> > > > > > > > >
>> >> > > > > > > > > Especially if we are looking at multiplexing streams - I would
>> >> > > > > wonder if
>> >> > > > > > > > > that's actually better served by making it easier to implement
>> >> > > > > using the
>> >> > > > > > > > > Flight implementation as it stands (by managing concurrent RPC
>> >> > > > > calls and/or
>> >> > > > > > > > > performing the union-of-structs encoding trick for you), instead
>> >> > > > > of having
>> >> > > > > > > > > to change the protocol.
>> >> > > > > > > > >
>> >> > > > > > > > > Nate: it may be worth starting a separate discussion about more
>> >> > > > > general
>> >> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
>> >> > > > > metadata was
>> >> > > > > > > > > chosen/if opaque bytes were considered in the past. Off the top
>> >> > > > of
>> >> > > > > my head
>> >> > > > > > > > > if it's for on-disk storage and fully application-defined it may
>> >> > > > > make sense
>> >> > > > > > > > > to store as a separate file alongside the Arrow file (indexed by
>> >> > > > > record
>> >> > > > > > > > > batch index) where you can take advantage of whatever format is
>> >> > > > > most
>> >> > > > > > > > > suitable.
>> >> > > > > > > > >
>> >> > > > > > > > > -David
>> >> > > > > > > > >
>> >> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
>> >> > > > > > > > > > Hi guys,
>> >> > > > > > > > > >
>> >> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
>> >> > > > to
>> >> > > > > add this
>> >> > > > > > > > > > feature starting from the IPC(I moved initial write up steps to
>> >> > > > > the
>> >> > > > > > > > > bottom
>> >> > > > > > > > > > of the doc). Afterwards David suggested focusing on Flight and
>> >> > > > > that's how
>> >> > > > > > > > > > we ended up with the protobufs change in the proposal. This
>> >> > > > > being said I
>> >> > > > > > > > > do
>> >> > > > > > > > > > think that the place where this should be impemented is a good
>> >> > > > > question
>> >> > > > > > > > > on
>> >> > > > > > > > > > its own. Maybe it makes sense to have this kind of a feature in
>> >> > > > > IPC and
>> >> > > > > > > > > > somehow use it in Flight, maybe not.
>> >> > > > > > > > > > 2. The point about dictionaries deserves a dedicated section in
>> >> > > > > the
>> >> > > > > > > > > > proposal. Nate and David brought it up and shared some
>> >> > > > insights.
>> >> > > > > I'll try
>> >> > > > > > > > > > to aggregate them and we can continue the discussion form
>> >> > > > there.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Cheers,
>> >> > > > > > > > > > Gosh
>> >> > > > > > > > > >
>> >> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
>> >> > > > > > > > > natebauernfeind@deephaven.io>
>> >> > > > > > > > > > wrote:
>> >> > > > > > > > > >
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > > > makes it more difficult to bring schema evolution back
>> >> > > > > into the
>> >> > > > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
>> >> > > > > > > > > > > > >
>> >> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
>> >> > > > > > > > > protobufs.
>> >> > > > > > > > > > > Can
>> >> > > > > > > > > > > > > you help me understand how difficult it would be to bring
>> >> > > > > the
>> >> > > > > > > > > > > `schema_id`
>> >> > > > > > > > > > > > > approach to the IPC stream format?
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > I thought we were talking solely about the Flight Protobuf
>> >> > > > > > > > > definitions -
>> >> > > > > > > > > > > > not the Flatbuffers (and the Google doc at least only talks
>> >> > > > > about the
>> >> > > > > > > > > > > > Protobufs).
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > I somehow missed that schema_id is being added to protobuf in
>> >> > > > > the
>> >> > > > > > > > > document.
>> >> > > > > > > > > > > It feels to me that the schema_id is a property that would
>> >> > > > > ideally only
>> >> > > > > > > > > > > apply to the RecordBatch. I better understand Micah's
>> >> > > > > dictionary
>> >> > > > > > > > > concerns,
>> >> > > > > > > > > > > now, too.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > > Side Question: Why isn't the IPC stream format a series of
>> >> > > > > the flight
>> >> > > > > > > > > > > > > protobufs? It's a real shame that there is no standard
>> >> > > > way
>> >> > > > > to
>> >> > > > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
>> >> > > > > ignoring the
>> >> > > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > > > The IPC format was defined long before Flight, and Flight's
>> >> > > > > > > > > app_metadata
>> >> > > > > > > > > > > > was added after Flight's initial definition. Note an IPC
>> >> > > > > message does
>> >> > > > > > > > > > > have
>> >> > > > > > > > > > > > a provision for key-value metadata, though I think APIs for
>> >> > > > > that are
>> >> > > > > > > > > not
>> >> > > > > > > > > > > > fully exposed. (See ARROW-6940:
>> >> > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
>> >> > > > > despite my
>> >> > > > > > > > > comments
>> >> > > > > > > > > > > > there perhaps we need to unify or at least consider how
>> >> > > > > Flight's
>> >> > > > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
>> >> > > > Also
>> >> > > > > > > > > perhaps see
>> >> > > > > > > > > > > > ARROW-1059.)
>> >> > > > > > > > > > > >
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
>> >> > > > > strings are
>> >> > > > > > > > > only
>> >> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
>> >> > > > > opaque
>> >> > > > > > > > > bytes.
>> >> > > > > > > > > > > The latter is a bit more useful.
>> >> > > > > > > > > > >
>> >> > > > > > > > > > > --
>> >> > > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > >

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

One issue with changing it to byte is it would effectively break any reader
that is validating flatbuffer data, because flatbuffers verifies null
termination [1].  While this might comply with forward compatibility
guarantees it seems like a pretty large blast radius.

[1]
https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flatbuffers.h#L2457

On Mon, Jul 12, 2021 at 12:38 PM Wes McKinney <we...@gmail.com> wrote:

> pyarrow at least treats the KeyValue values as binary and not UTF-8.
>
> On Sun, Jul 11, 2021 at 9:40 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >
> > I think other languages (e.g. java, python) might make more of
> distinction between utf-8 compatible strings and raw bytes.  For python it
> might be less of a concern if the c++ wrapper already makes the value field
> look like a bytes field
> >
> > On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:
> >>
> >> We could certainly "upgrade" KeyValue to have a binary value field
> >> everywhere KeyValue is used, but there is some risk of code in the
> >> wild expecting there to be a null terminator after the string data.
> >> The Flatbuffers-generated accessor APIs do not depend on the existence
> >> of the null terminator, though. Not ideal, but I would not be thrilled
> >> about adding an extra [ BinaryKeyValue ] everyplace we currently have
> >> [ KeyValue ].
> >>
> >> That said, I doubt that we have any endogenous forward compatibility
> >> problems related to this in Apache Arrow-maintained libraries, the
> >> risk would come from users who are interacting with the Flatbuffers
> >> data manually / without using one of our libraries. We could implement
> >> the changes and run a set of forward compatibility integration tests
> >> to see if anyone of our released libraries have an issue.
> >>
> >> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com>
> wrote:
> >> >
> >> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
> >> >
> >> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <em...@gmail.com>
> wrote:
> >> > >
> >> > > Retitling and forking the discussion to talk about key value pairs.
> >> > >
> >> > > What is the byte cost of an empty list?  Another option would be to
> >> > > introduce a new BinaryKeyValue table and add binary metadata.
> >> > >
> >> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> >> > > natebauernfeind@deephaven.io> wrote:
> >> > >
> >> > > > Deephaven and I are very supportive of "upgrading" the value half
> of the kv
> >> > > > pair to a byte vector. What is the best way to find out if there
> is
> >> > > > sufficient interest?
> >> > > >
> >> > > >
> >> > > > I've been stewing on the ideas here around schema evolution, and
> I realize
> >> > > > the specific feature I am missing is the ability to encode that a
> field
> >> > > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch)
> is
> >> > > > empty/has-no-data in O(0) cost (yes; for free).
> >> > > >
> >> > > > Might there be interest in adding a "field_id" to the FieldNode
> (which is
> >> > > > encoded on the RecordBatch flatbuffer)? I see a simple
> forward-compatible
> >> > > > upgrade (by either keying off of 0, or explicitly set the field
> default to
> >> > > > -1) which would allow the sender to "skip" fields that have 1)
> FieldNode
> >> > > > length of zero, and 2) all Buffer's associated at that level (and
> further
> >> > > > nested) are also equally empty (i.e. Buffer length is zero).
> >> > > >
> >> > > > I understand this concept slightly interferes with RecordBatch's
> `length`
> >> > > > field, and that many implementations use that length to resize the
> >> > > > root-level FieldNodes. The use-case I have in mind has different
> logical
> >> > > > lengths per field node; current implementations require sending a
> >> > > > RecordBatch length of the max length across all root level field
> nodes. I
> >> > > > believe this requires a copy of data whenever a field node is too
> short; I
> >> > > > don't know if there is a decent solution to this slight
> inefficiency. I am
> >> > > > bringing it up because if "skipping a field node when it is
> empty" is a
> >> > > > feature, then we may not want to allocate space for those nodes
> given that
> >> > > > the record batch length will likely be greater than zero.
> >> > > >
> >> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com>
> wrote:
> >> > > >
> >> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me>
> wrote:
> >> > > > > >
> >> > > > > > From the Flatbuffers internals doc[1] it appears they are the
> same:
> >> > > > > "Strings are simply a vector of bytes, and are always
> null-terminated."
> >> > > > >
> >> > > > > I see. I took a look at flatbuffers.h, and it appears that
> changing
> >> > > > > this field from string to [byte] would be backward-compatible
> and
> >> > > > > forward-compatible except with code that expects a null
> terminator.
> >> > > > > This is something we could discuss separately if there were
> enough
> >> > > > > interest.
> >> > > > >
> >> > > > > > [1]:
> https://google.github.io/flatbuffers/flatbuffers_internals.html
> >> > > > > >
> >> > > > > > -David
> >> > > > > >
> >> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> >> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> >> > > > emkornfield@gmail.com>
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Right, I had wanted to focus the discussion on Flight
> as I think
> >> > > > > schema
> >> > > > > > > > > evolution or multiplexing streams (more so the latter)
> is a
> >> > > > > property of the
> >> > > > > > > > > transport and not the stream format itself. If we are
> leaning
> >> > > > > towards just
> >> > > > > > > > > schema evolution then maybe it makes sense to discuss
> it for the
> >> > > > > IPC stream
> >> > > > > > > > > format and leverage that in Flight. I'd be interested
> in what
> >> > > > > others think.
> >> > > > > > > >
> >> > > > > > > > I tend to agree, I think stream multiplexing is likely a
> transport
> >> > > > > level
> >> > > > > > > > issue.  IMO I think schema evolution should be consistent
> with the
> >> > > > > IPC
> >> > > > > > > > stream format  and flight.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > > Nate: it may be worth starting a separate discussion
> about more
> >> > > > > general
> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> key-value
> >> > > > > metadata was
> >> > > > > > > > > chosen/if opaque bytes were considered in the past.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I think  this was an unfortunate design of the key value
> metadata
> >> > > > in
> >> > > > > > > > Schema.fbs, but I don't think I was around when this
> decision was
> >> > > > > made.
> >> > > > > > >
> >> > > > > > > I agree that it's unfortunate that we did not use [ byte ]
> instead of
> >> > > > > > > string for the value in the KeyValue metadata — I think
> this was more
> >> > > > > > > of an oversight than a deliberate choice (e.g. it was not
> our intent
> >> > > > > > > to require binary data to be base64-encoded — this is
> something that
> >> > > > > > > we have to do when encoding binary data in Thrift KeyValue
> metadata
> >> > > > > > > for Parquet, for example). Is the binary representation of
> [byte]
> >> > > > > > > different from string?
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > > Side Question: Why isn't the IPC stream format a series
> of the
> >> > > > flight
> >> > > > > > > > > protobufs?
> >> > > > > > > >
> >> > > > > > > > In addition to what David said, protobufs can't be read
> directly
> >> > > > > from a
> >> > > > > > > > memory-mapped file (they need decoding).  This was one of
> the
> >> > > > design
> >> > > > > > > > considerations of using flatbuffers and IPC Stream/File
> format.
> >> > > > > > > >
> >> > > > > > > > I was thinking Micah's comment is more that whatever we
> do, it
> >> > > > > should be
> >> > > > > > > > > clearly specified and edge cases should be considered,
> especially
> >> > > > > if we
> >> > > > > > > > > might want to 'backport' this into the stream format
> later.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Yes, for dictionaries we just need to be careful to define
> >> > > > semantics
> >> > > > > and
> >> > > > > > > > ensure implementations are validating them with regards to
> >> > > > > dictionaries.
> >> > > > > > > > There likely isn't any need to change current
> implementations
> >> > > > though.
> >> > > > > > > >
> >> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <
> lidavidm@apache.org>
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Right, I had wanted to focus the discussion on Flight
> as I think
> >> > > > > schema
> >> > > > > > > > > evolution or multiplexing streams (more so the latter)
> is a
> >> > > > > property of the
> >> > > > > > > > > transport and not the stream format itself. If we are
> leaning
> >> > > > > towards just
> >> > > > > > > > > schema evolution then maybe it makes sense to discuss
> it for the
> >> > > > > IPC stream
> >> > > > > > > > > format and leverage that in Flight. I'd be interested
> in what
> >> > > > > others think.
> >> > > > > > > > >
> >> > > > > > > > > Especially if we are looking at multiplexing streams -
> I would
> >> > > > > wonder if
> >> > > > > > > > > that's actually better served by making it easier to
> implement
> >> > > > > using the
> >> > > > > > > > > Flight implementation as it stands (by managing
> concurrent RPC
> >> > > > > calls and/or
> >> > > > > > > > > performing the union-of-structs encoding trick for
> you), instead
> >> > > > > of having
> >> > > > > > > > > to change the protocol.
> >> > > > > > > > >
> >> > > > > > > > > Nate: it may be worth starting a separate discussion
> about more
> >> > > > > general
> >> > > > > > > > > metadata in the IPC message. I'm not aware of why
> key-value
> >> > > > > metadata was
> >> > > > > > > > > chosen/if opaque bytes were considered in the past. Off
> the top
> >> > > > of
> >> > > > > my head
> >> > > > > > > > > if it's for on-disk storage and fully
> application-defined it may
> >> > > > > make sense
> >> > > > > > > > > to store as a separate file alongside the Arrow file
> (indexed by
> >> > > > > record
> >> > > > > > > > > batch index) where you can take advantage of whatever
> format is
> >> > > > > most
> >> > > > > > > > > suitable.
> >> > > > > > > > >
> >> > > > > > > > > -David
> >> > > > > > > > >
> >> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> >> > > > > > > > > > Hi guys,
> >> > > > > > > > > >
> >> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial
> suggestion was
> >> > > > to
> >> > > > > add this
> >> > > > > > > > > > feature starting from the IPC(I moved initial write
> up steps to
> >> > > > > the
> >> > > > > > > > > bottom
> >> > > > > > > > > > of the doc). Afterwards David suggested focusing on
> Flight and
> >> > > > > that's how
> >> > > > > > > > > > we ended up with the protobufs change in the
> proposal. This
> >> > > > > being said I
> >> > > > > > > > > do
> >> > > > > > > > > > think that the place where this should be impemented
> is a good
> >> > > > > question
> >> > > > > > > > > on
> >> > > > > > > > > > its own. Maybe it makes sense to have this kind of a
> feature in
> >> > > > > IPC and
> >> > > > > > > > > > somehow use it in Flight, maybe not.
> >> > > > > > > > > > 2. The point about dictionaries deserves a dedicated
> section in
> >> > > > > the
> >> > > > > > > > > > proposal. Nate and David brought it up and shared some
> >> > > > insights.
> >> > > > > I'll try
> >> > > > > > > > > > to aggregate them and we can continue the discussion
> form
> >> > > > there.
> >> > > > > > > > > >
> >> > > > > > > > > > Cheers,
> >> > > > > > > > > > Gosh
> >> > > > > > > > > >
> >> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> >> > > > > > > > > natebauernfeind@deephaven.io>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > > makes it more difficult to bring schema
> evolution back
> >> > > > > into the
> >> > > > > > > > > > > > > > IPC Stream format (i.e. it would live only in
> flight)
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer
> structures not the
> >> > > > > > > > > protobufs.
> >> > > > > > > > > > > Can
> >> > > > > > > > > > > > > you help me understand how difficult it would
> be to bring
> >> > > > > the
> >> > > > > > > > > > > `schema_id`
> >> > > > > > > > > > > > > approach to the IPC stream format?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I thought we were talking solely about the Flight
> Protobuf
> >> > > > > > > > > definitions -
> >> > > > > > > > > > > > not the Flatbuffers (and the Google doc at least
> only talks
> >> > > > > about the
> >> > > > > > > > > > > > Protobufs).
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > I somehow missed that schema_id is being added to
> protobuf in
> >> > > > > the
> >> > > > > > > > > document.
> >> > > > > > > > > > > It feels to me that the schema_id is a property
> that would
> >> > > > > ideally only
> >> > > > > > > > > > > apply to the RecordBatch. I better understand
> Micah's
> >> > > > > dictionary
> >> > > > > > > > > concerns,
> >> > > > > > > > > > > now, too.
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Side Question: Why isn't the IPC stream format a
> series of
> >> > > > > the flight
> >> > > > > > > > > > > > > protobufs? It's a real shame that there is no
> standard
> >> > > > way
> >> > > > > to
> >> > > > > > > > > > > > > capture/replay a stream with app_metadata.
> (Obviously
> >> > > > > ignoring the
> >> > > > > > > > > > > > > annoyances around protobuf wrapping
> flatbuffers.)
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > The IPC format was defined long before Flight,
> and Flight's
> >> > > > > > > > > app_metadata
> >> > > > > > > > > > > > was added after Flight's initial definition. Note
> an IPC
> >> > > > > message does
> >> > > > > > > > > > > have
> >> > > > > > > > > > > > a provision for key-value metadata, though I
> think APIs for
> >> > > > > that are
> >> > > > > > > > > not
> >> > > > > > > > > > > > fully exposed. (See ARROW-6940:
> >> > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940
> and
> >> > > > > despite my
> >> > > > > > > > > comments
> >> > > > > > > > > > > > there perhaps we need to unify or at least
> consider how
> >> > > > > Flight's
> >> > > > > > > > > > > > app_metadata relates to the IPC message
> custom_metadata.
> >> > > > Also
> >> > > > > > > > > perhaps see
> >> > > > > > > > > > > > ARROW-1059.)
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > KeyValue unfortunately is string to string. In
> flatbuffer
> >> > > > > strings are
> >> > > > > > > > > only
> >> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other
> hand is
> >> > > > > opaque
> >> > > > > > > > > bytes.
> >> > > > > > > > > > > The latter is a bit more useful.
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
>

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

pyarrow at least treats the KeyValue values as binary and not UTF-8.

On Sun, Jul 11, 2021 at 9:40 PM Micah Kornfield <em...@gmail.com> wrote:
>
> I think other languages (e.g. java, python) might make more of distinction between utf-8 compatible strings and raw bytes.  For python it might be less of a concern if the c++ wrapper already makes the value field look like a bytes field
>
> On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:
>>
>> We could certainly "upgrade" KeyValue to have a binary value field
>> everywhere KeyValue is used, but there is some risk of code in the
>> wild expecting there to be a null terminator after the string data.
>> The Flatbuffers-generated accessor APIs do not depend on the existence
>> of the null terminator, though. Not ideal, but I would not be thrilled
>> about adding an extra [ BinaryKeyValue ] everyplace we currently have
>> [ KeyValue ].
>>
>> That said, I doubt that we have any endogenous forward compatibility
>> problems related to this in Apache Arrow-maintained libraries, the
>> risk would come from users who are interacting with the Flatbuffers
>> data manually / without using one of our libraries. We could implement
>> the changes and run a set of forward compatibility integration tests
>> to see if anyone of our released libraries have an issue.
>>
>> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com> wrote:
>> >
>> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
>> >
>> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <em...@gmail.com> wrote:
>> > >
>> > > Retitling and forking the discussion to talk about key value pairs.
>> > >
>> > > What is the byte cost of an empty list?  Another option would be to
>> > > introduce a new BinaryKeyValue table and add binary metadata.
>> > >
>> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
>> > > natebauernfeind@deephaven.io> wrote:
>> > >
>> > > > Deephaven and I are very supportive of "upgrading" the value half of the kv
>> > > > pair to a byte vector. What is the best way to find out if there is
>> > > > sufficient interest?
>> > > >
>> > > >
>> > > > I've been stewing on the ideas here around schema evolution, and I realize
>> > > > the specific feature I am missing is the ability to encode that a field
>> > > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
>> > > > empty/has-no-data in O(0) cost (yes; for free).
>> > > >
>> > > > Might there be interest in adding a "field_id" to the FieldNode (which is
>> > > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
>> > > > upgrade (by either keying off of 0, or explicitly set the field default to
>> > > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
>> > > > length of zero, and 2) all Buffer's associated at that level (and further
>> > > > nested) are also equally empty (i.e. Buffer length is zero).
>> > > >
>> > > > I understand this concept slightly interferes with RecordBatch's `length`
>> > > > field, and that many implementations use that length to resize the
>> > > > root-level FieldNodes. The use-case I have in mind has different logical
>> > > > lengths per field node; current implementations require sending a
>> > > > RecordBatch length of the max length across all root level field nodes. I
>> > > > believe this requires a copy of data whenever a field node is too short; I
>> > > > don't know if there is a decent solution to this slight inefficiency. I am
>> > > > bringing it up because if "skipping a field node when it is empty" is a
>> > > > feature, then we may not want to allocate space for those nodes given that
>> > > > the record batch length will likely be greater than zero.
>> > > >
>> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
>> > > >
>> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
>> > > > > >
>> > > > > > From the Flatbuffers internals doc[1] it appears they are the same:
>> > > > > "Strings are simply a vector of bytes, and are always null-terminated."
>> > > > >
>> > > > > I see. I took a look at flatbuffers.h, and it appears that changing
>> > > > > this field from string to [byte] would be backward-compatible and
>> > > > > forward-compatible except with code that expects a null terminator.
>> > > > > This is something we could discuss separately if there were enough
>> > > > > interest.
>> > > > >
>> > > > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
>> > > > > >
>> > > > > > -David
>> > > > > >
>> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
>> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
>> > > > emkornfield@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
>> > > > > schema
>> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
>> > > > > property of the
>> > > > > > > > > transport and not the stream format itself. If we are leaning
>> > > > > towards just
>> > > > > > > > > schema evolution then maybe it makes sense to discuss it for the
>> > > > > IPC stream
>> > > > > > > > > format and leverage that in Flight. I'd be interested in what
>> > > > > others think.
>> > > > > > > >
>> > > > > > > > I tend to agree, I think stream multiplexing is likely a transport
>> > > > > level
>> > > > > > > > issue.  IMO I think schema evolution should be consistent with the
>> > > > > IPC
>> > > > > > > > stream format  and flight.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > > Nate: it may be worth starting a separate discussion about more
>> > > > > general
>> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
>> > > > > metadata was
>> > > > > > > > > chosen/if opaque bytes were considered in the past.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I think  this was an unfortunate design of the key value metadata
>> > > > in
>> > > > > > > > Schema.fbs, but I don't think I was around when this decision was
>> > > > > made.
>> > > > > > >
>> > > > > > > I agree that it's unfortunate that we did not use [ byte ] instead of
>> > > > > > > string for the value in the KeyValue metadata — I think this was more
>> > > > > > > of an oversight than a deliberate choice (e.g. it was not our intent
>> > > > > > > to require binary data to be base64-encoded — this is something that
>> > > > > > > we have to do when encoding binary data in Thrift KeyValue metadata
>> > > > > > > for Parquet, for example). Is the binary representation of [byte]
>> > > > > > > different from string?
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > > Side Question: Why isn't the IPC stream format a series of the
>> > > > flight
>> > > > > > > > > protobufs?
>> > > > > > > >
>> > > > > > > > In addition to what David said, protobufs can't be read directly
>> > > > > from a
>> > > > > > > > memory-mapped file (they need decoding).  This was one of the
>> > > > design
>> > > > > > > > considerations of using flatbuffers and IPC Stream/File format.
>> > > > > > > >
>> > > > > > > > I was thinking Micah's comment is more that whatever we do, it
>> > > > > should be
>> > > > > > > > > clearly specified and edge cases should be considered, especially
>> > > > > if we
>> > > > > > > > > might want to 'backport' this into the stream format later.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Yes, for dictionaries we just need to be careful to define
>> > > > semantics
>> > > > > and
>> > > > > > > > ensure implementations are validating them with regards to
>> > > > > dictionaries.
>> > > > > > > > There likely isn't any need to change current implementations
>> > > > though.
>> > > > > > > >
>> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
>> > > > > schema
>> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
>> > > > > property of the
>> > > > > > > > > transport and not the stream format itself. If we are leaning
>> > > > > towards just
>> > > > > > > > > schema evolution then maybe it makes sense to discuss it for the
>> > > > > IPC stream
>> > > > > > > > > format and leverage that in Flight. I'd be interested in what
>> > > > > others think.
>> > > > > > > > >
>> > > > > > > > > Especially if we are looking at multiplexing streams - I would
>> > > > > wonder if
>> > > > > > > > > that's actually better served by making it easier to implement
>> > > > > using the
>> > > > > > > > > Flight implementation as it stands (by managing concurrent RPC
>> > > > > calls and/or
>> > > > > > > > > performing the union-of-structs encoding trick for you), instead
>> > > > > of having
>> > > > > > > > > to change the protocol.
>> > > > > > > > >
>> > > > > > > > > Nate: it may be worth starting a separate discussion about more
>> > > > > general
>> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
>> > > > > metadata was
>> > > > > > > > > chosen/if opaque bytes were considered in the past. Off the top
>> > > > of
>> > > > > my head
>> > > > > > > > > if it's for on-disk storage and fully application-defined it may
>> > > > > make sense
>> > > > > > > > > to store as a separate file alongside the Arrow file (indexed by
>> > > > > record
>> > > > > > > > > batch index) where you can take advantage of whatever format is
>> > > > > most
>> > > > > > > > > suitable.
>> > > > > > > > >
>> > > > > > > > > -David
>> > > > > > > > >
>> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
>> > > > > > > > > > Hi guys,
>> > > > > > > > > >
>> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
>> > > > to
>> > > > > add this
>> > > > > > > > > > feature starting from the IPC(I moved initial write up steps to
>> > > > > the
>> > > > > > > > > bottom
>> > > > > > > > > > of the doc). Afterwards David suggested focusing on Flight and
>> > > > > that's how
>> > > > > > > > > > we ended up with the protobufs change in the proposal. This
>> > > > > being said I
>> > > > > > > > > do
>> > > > > > > > > > think that the place where this should be impemented is a good
>> > > > > question
>> > > > > > > > > on
>> > > > > > > > > > its own. Maybe it makes sense to have this kind of a feature in
>> > > > > IPC and
>> > > > > > > > > > somehow use it in Flight, maybe not.
>> > > > > > > > > > 2. The point about dictionaries deserves a dedicated section in
>> > > > > the
>> > > > > > > > > > proposal. Nate and David brought it up and shared some
>> > > > insights.
>> > > > > I'll try
>> > > > > > > > > > to aggregate them and we can continue the discussion form
>> > > > there.
>> > > > > > > > > >
>> > > > > > > > > > Cheers,
>> > > > > > > > > > Gosh
>> > > > > > > > > >
>> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
>> > > > > > > > > natebauernfeind@deephaven.io>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > > > makes it more difficult to bring schema evolution back
>> > > > > into the
>> > > > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
>> > > > > > > > > protobufs.
>> > > > > > > > > > > Can
>> > > > > > > > > > > > > you help me understand how difficult it would be to bring
>> > > > > the
>> > > > > > > > > > > `schema_id`
>> > > > > > > > > > > > > approach to the IPC stream format?
>> > > > > > > > > > > >
>> > > > > > > > > > > > I thought we were talking solely about the Flight Protobuf
>> > > > > > > > > definitions -
>> > > > > > > > > > > > not the Flatbuffers (and the Google doc at least only talks
>> > > > > about the
>> > > > > > > > > > > > Protobufs).
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > I somehow missed that schema_id is being added to protobuf in
>> > > > > the
>> > > > > > > > > document.
>> > > > > > > > > > > It feels to me that the schema_id is a property that would
>> > > > > ideally only
>> > > > > > > > > > > apply to the RecordBatch. I better understand Micah's
>> > > > > dictionary
>> > > > > > > > > concerns,
>> > > > > > > > > > > now, too.
>> > > > > > > > > > >
>> > > > > > > > > > > > Side Question: Why isn't the IPC stream format a series of
>> > > > > the flight
>> > > > > > > > > > > > > protobufs? It's a real shame that there is no standard
>> > > > way
>> > > > > to
>> > > > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
>> > > > > ignoring the
>> > > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
>> > > > > > > > > > > >
>> > > > > > > > > > > > The IPC format was defined long before Flight, and Flight's
>> > > > > > > > > app_metadata
>> > > > > > > > > > > > was added after Flight's initial definition. Note an IPC
>> > > > > message does
>> > > > > > > > > > > have
>> > > > > > > > > > > > a provision for key-value metadata, though I think APIs for
>> > > > > that are
>> > > > > > > > > not
>> > > > > > > > > > > > fully exposed. (See ARROW-6940:
>> > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
>> > > > > despite my
>> > > > > > > > > comments
>> > > > > > > > > > > > there perhaps we need to unify or at least consider how
>> > > > > Flight's
>> > > > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
>> > > > Also
>> > > > > > > > > perhaps see
>> > > > > > > > > > > > ARROW-1059.)
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
>> > > > > strings are
>> > > > > > > > > only
>> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
>> > > > > opaque
>> > > > > > > > > bytes.
>> > > > > > > > > > > The latter is a bit more useful.
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > >

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

I think other languages (e.g. java, python) might make more of distinction
between utf-8 compatible strings and raw bytes.  For python it might be
less of a concern if the c++ wrapper already makes the value field look
like a bytes field

On Sunday, July 11, 2021, Wes McKinney <we...@gmail.com> wrote:

> We could certainly "upgrade" KeyValue to have a binary value field
> everywhere KeyValue is used, but there is some risk of code in the
> wild expecting there to be a null terminator after the string data.
> The Flatbuffers-generated accessor APIs do not depend on the existence
> of the null terminator, though. Not ideal, but I would not be thrilled
> about adding an extra [ BinaryKeyValue ] everyplace we currently have
> [ KeyValue ].
>
> That said, I doubt that we have any endogenous forward compatibility
> problems related to this in Apache Arrow-maintained libraries, the
> risk would come from users who are interacting with the Flatbuffers
> data manually / without using one of our libraries. We could implement
> the changes and run a set of forward compatibility integration tests
> to see if anyone of our released libraries have an issue.
>
> On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > The cost of an empty vector in Flatbuffers appears to be 4 bytes.
> >
> > On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <em...@gmail.com>
> wrote:
> > >
> > > Retitling and forking the discussion to talk about key value pairs.
> > >
> > > What is the byte cost of an empty list?  Another option would be to
> > > introduce a new BinaryKeyValue table and add binary metadata.
> > >
> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > > natebauernfeind@deephaven.io> wrote:
> > >
> > > > Deephaven and I are very supportive of "upgrading" the value half of
> the kv
> > > > pair to a byte vector. What is the best way to find out if there is
> > > > sufficient interest?
> > > >
> > > >
> > > > I've been stewing on the ideas here around schema evolution, and I
> realize
> > > > the specific feature I am missing is the ability to encode that a
> field
> > > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > > > empty/has-no-data in O(0) cost (yes; for free).
> > > >
> > > > Might there be interest in adding a "field_id" to the FieldNode
> (which is
> > > > encoded on the RecordBatch flatbuffer)? I see a simple
> forward-compatible
> > > > upgrade (by either keying off of 0, or explicitly set the field
> default to
> > > > -1) which would allow the sender to "skip" fields that have 1)
> FieldNode
> > > > length of zero, and 2) all Buffer's associated at that level (and
> further
> > > > nested) are also equally empty (i.e. Buffer length is zero).
> > > >
> > > > I understand this concept slightly interferes with RecordBatch's
> `length`
> > > > field, and that many implementations use that length to resize the
> > > > root-level FieldNodes. The use-case I have in mind has different
> logical
> > > > lengths per field node; current implementations require sending a
> > > > RecordBatch length of the max length across all root level field
> nodes. I
> > > > believe this requires a copy of data whenever a field node is too
> short; I
> > > > don't know if there is a decent solution to this slight
> inefficiency. I am
> > > > bringing it up because if "skipping a field node when it is empty"
> is a
> > > > feature, then we may not want to allocate space for those nodes
> given that
> > > > the record batch length will likely be greater than zero.
> > > >
> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com>
> wrote:
> > > >
> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me>
> wrote:
> > > > > >
> > > > > > From the Flatbuffers internals doc[1] it appears they are the
> same:
> > > > > "Strings are simply a vector of bytes, and are always
> null-terminated."
> > > > >
> > > > > I see. I took a look at flatbuffers.h, and it appears that changing
> > > > > this field from string to [byte] would be backward-compatible and
> > > > > forward-compatible except with code that expects a null terminator.
> > > > > This is something we could discuss separately if there were enough
> > > > > interest.
> > > > >
> > > > > > [1]: https://google.github.io/flatbuffers/flatbuffers_
> internals.html
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > > > emkornfield@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I
> think
> > > > > schema
> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > > property of the
> > > > > > > > > transport and not the stream format itself. If we are
> leaning
> > > > > towards just
> > > > > > > > > schema evolution then maybe it makes sense to discuss it
> for the
> > > > > IPC stream
> > > > > > > > > format and leverage that in Flight. I'd be interested in
> what
> > > > > others think.
> > > > > > > >
> > > > > > > > I tend to agree, I think stream multiplexing is likely a
> transport
> > > > > level
> > > > > > > > issue.  IMO I think schema evolution should be consistent
> with the
> > > > > IPC
> > > > > > > > stream format  and flight.
> > > > > > > >
> > > > > > > >
> > > > > > > > > Nate: it may be worth starting a separate discussion about
> more
> > > > > general
> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > > metadata was
> > > > > > > > > chosen/if opaque bytes were considered in the past.
> > > > > > > >
> > > > > > > >
> > > > > > > > I think  this was an unfortunate design of the key value
> metadata
> > > > in
> > > > > > > > Schema.fbs, but I don't think I was around when this
> decision was
> > > > > made.
> > > > > > >
> > > > > > > I agree that it's unfortunate that we did not use [ byte ]
> instead of
> > > > > > > string for the value in the KeyValue metadata — I think this
> was more
> > > > > > > of an oversight than a deliberate choice (e.g. it was not our
> intent
> > > > > > > to require binary data to be base64-encoded — this is
> something that
> > > > > > > we have to do when encoding binary data in Thrift KeyValue
> metadata
> > > > > > > for Parquet, for example). Is the binary representation of
> [byte]
> > > > > > > different from string?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Side Question: Why isn't the IPC stream format a series of
> the
> > > > flight
> > > > > > > > > protobufs?
> > > > > > > >
> > > > > > > > In addition to what David said, protobufs can't be read
> directly
> > > > > from a
> > > > > > > > memory-mapped file (they need decoding).  This was one of the
> > > > design
> > > > > > > > considerations of using flatbuffers and IPC Stream/File
> format.
> > > > > > > >
> > > > > > > > I was thinking Micah's comment is more that whatever we do,
> it
> > > > > should be
> > > > > > > > > clearly specified and edge cases should be considered,
> especially
> > > > > if we
> > > > > > > > > might want to 'backport' this into the stream format later.
> > > > > > > >
> > > > > > > >
> > > > > > > > Yes, for dictionaries we just need to be careful to define
> > > > semantics
> > > > > and
> > > > > > > > ensure implementations are validating them with regards to
> > > > > dictionaries.
> > > > > > > > There likely isn't any need to change current implementations
> > > > though.
> > > > > > > >
> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <
> lidavidm@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I
> think
> > > > > schema
> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > > property of the
> > > > > > > > > transport and not the stream format itself. If we are
> leaning
> > > > > towards just
> > > > > > > > > schema evolution then maybe it makes sense to discuss it
> for the
> > > > > IPC stream
> > > > > > > > > format and leverage that in Flight. I'd be interested in
> what
> > > > > others think.
> > > > > > > > >
> > > > > > > > > Especially if we are looking at multiplexing streams - I
> would
> > > > > wonder if
> > > > > > > > > that's actually better served by making it easier to
> implement
> > > > > using the
> > > > > > > > > Flight implementation as it stands (by managing concurrent
> RPC
> > > > > calls and/or
> > > > > > > > > performing the union-of-structs encoding trick for you),
> instead
> > > > > of having
> > > > > > > > > to change the protocol.
> > > > > > > > >
> > > > > > > > > Nate: it may be worth starting a separate discussion about
> more
> > > > > general
> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > > metadata was
> > > > > > > > > chosen/if opaque bytes were considered in the past. Off
> the top
> > > > of
> > > > > my head
> > > > > > > > > if it's for on-disk storage and fully application-defined
> it may
> > > > > make sense
> > > > > > > > > to store as a separate file alongside the Arrow file
> (indexed by
> > > > > record
> > > > > > > > > batch index) where you can take advantage of whatever
> format is
> > > > > most
> > > > > > > > > suitable.
> > > > > > > > >
> > > > > > > > > -David
> > > > > > > > >
> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > > > > Hi guys,
> > > > > > > > > >
> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial
> suggestion was
> > > > to
> > > > > add this
> > > > > > > > > > feature starting from the IPC(I moved initial write up
> steps to
> > > > > the
> > > > > > > > > bottom
> > > > > > > > > > of the doc). Afterwards David suggested focusing on
> Flight and
> > > > > that's how
> > > > > > > > > > we ended up with the protobufs change in the proposal.
> This
> > > > > being said I
> > > > > > > > > do
> > > > > > > > > > think that the place where this should be impemented is
> a good
> > > > > question
> > > > > > > > > on
> > > > > > > > > > its own. Maybe it makes sense to have this kind of a
> feature in
> > > > > IPC and
> > > > > > > > > > somehow use it in Flight, maybe not.
> > > > > > > > > > 2. The point about dictionaries deserves a dedicated
> section in
> > > > > the
> > > > > > > > > > proposal. Nate and David brought it up and shared some
> > > > insights.
> > > > > I'll try
> > > > > > > > > > to aggregate them and we can continue the discussion form
> > > > there.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Gosh
> > > > > > > > > >
> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > > > > natebauernfeind@deephaven.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > > makes it more difficult to bring schema
> evolution back
> > > > > into the
> > > > > > > > > > > > > > IPC Stream format (i.e. it would live only in
> flight)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures
> not the
> > > > > > > > > protobufs.
> > > > > > > > > > > Can
> > > > > > > > > > > > > you help me understand how difficult it would be
> to bring
> > > > > the
> > > > > > > > > > > `schema_id`
> > > > > > > > > > > > > approach to the IPC stream format?
> > > > > > > > > > > >
> > > > > > > > > > > > I thought we were talking solely about the Flight
> Protobuf
> > > > > > > > > definitions -
> > > > > > > > > > > > not the Flatbuffers (and the Google doc at least
> only talks
> > > > > about the
> > > > > > > > > > > > Protobufs).
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I somehow missed that schema_id is being added to
> protobuf in
> > > > > the
> > > > > > > > > document.
> > > > > > > > > > > It feels to me that the schema_id is a property that
> would
> > > > > ideally only
> > > > > > > > > > > apply to the RecordBatch. I better understand Micah's
> > > > > dictionary
> > > > > > > > > concerns,
> > > > > > > > > > > now, too.
> > > > > > > > > > >
> > > > > > > > > > > > Side Question: Why isn't the IPC stream format a
> series of
> > > > > the flight
> > > > > > > > > > > > > protobufs? It's a real shame that there is no
> standard
> > > > way
> > > > > to
> > > > > > > > > > > > > capture/replay a stream with app_metadata.
> (Obviously
> > > > > ignoring the
> > > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > > > > >
> > > > > > > > > > > > The IPC format was defined long before Flight, and
> Flight's
> > > > > > > > > app_metadata
> > > > > > > > > > > > was added after Flight's initial definition. Note an
> IPC
> > > > > message does
> > > > > > > > > > > have
> > > > > > > > > > > > a provision for key-value metadata, though I think
> APIs for
> > > > > that are
> > > > > > > > > not
> > > > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > > > despite my
> > > > > > > > > comments
> > > > > > > > > > > > there perhaps we need to unify or at least consider
> how
> > > > > Flight's
> > > > > > > > > > > > app_metadata relates to the IPC message
> custom_metadata.
> > > > Also
> > > > > > > > > perhaps see
> > > > > > > > > > > > ARROW-1059.)
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > KeyValue unfortunately is string to string. In
> flatbuffer
> > > > > strings are
> > > > > > > > > only
> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other
> hand is
> > > > > opaque
> > > > > > > > > bytes.
> > > > > > > > > > > The latter is a bit more useful.
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
>

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

We could certainly "upgrade" KeyValue to have a binary value field
everywhere KeyValue is used, but there is some risk of code in the
wild expecting there to be a null terminator after the string data.
The Flatbuffers-generated accessor APIs do not depend on the existence
of the null terminator, though. Not ideal, but I would not be thrilled
about adding an extra [ BinaryKeyValue ] everyplace we currently have
[ KeyValue ].

That said, I doubt that we have any endogenous forward compatibility
problems related to this in Apache Arrow-maintained libraries, the
risk would come from users who are interacting with the Flatbuffers
data manually / without using one of our libraries. We could implement
the changes and run a set of forward compatibility integration tests
to see if anyone of our released libraries have an issue.

On Fri, Jul 9, 2021 at 11:33 AM Wes McKinney <we...@gmail.com> wrote:
>
> The cost of an empty vector in Flatbuffers appears to be 4 bytes.
>
> On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <em...@gmail.com> wrote:
> >
> > Retitling and forking the discussion to talk about key value pairs.
> >
> > What is the byte cost of an empty list?  Another option would be to
> > introduce a new BinaryKeyValue table and add binary metadata.
> >
> > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > natebauernfeind@deephaven.io> wrote:
> >
> > > Deephaven and I are very supportive of "upgrading" the value half of the kv
> > > pair to a byte vector. What is the best way to find out if there is
> > > sufficient interest?
> > >
> > >
> > > I've been stewing on the ideas here around schema evolution, and I realize
> > > the specific feature I am missing is the ability to encode that a field
> > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > > empty/has-no-data in O(0) cost (yes; for free).
> > >
> > > Might there be interest in adding a "field_id" to the FieldNode (which is
> > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > > upgrade (by either keying off of 0, or explicitly set the field default to
> > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > > length of zero, and 2) all Buffer's associated at that level (and further
> > > nested) are also equally empty (i.e. Buffer length is zero).
> > >
> > > I understand this concept slightly interferes with RecordBatch's `length`
> > > field, and that many implementations use that length to resize the
> > > root-level FieldNodes. The use-case I have in mind has different logical
> > > lengths per field node; current implementations require sending a
> > > RecordBatch length of the max length across all root level field nodes. I
> > > believe this requires a copy of data whenever a field node is too short; I
> > > don't know if there is a decent solution to this slight inefficiency. I am
> > > bringing it up because if "skipping a field node when it is empty" is a
> > > feature, then we may not want to allocate space for those nodes given that
> > > the record batch length will likely be greater than zero.
> > >
> > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> > > > >
> > > > > From the Flatbuffers internals doc[1] it appears they are the same:
> > > > "Strings are simply a vector of bytes, and are always null-terminated."
> > > >
> > > > I see. I took a look at flatbuffers.h, and it appears that changing
> > > > this field from string to [byte] would be backward-compatible and
> > > > forward-compatible except with code that expects a null terminator.
> > > > This is something we could discuss separately if there were enough
> > > > interest.
> > > >
> > > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > > >
> > > > > -David
> > > > >
> > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > > emkornfield@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > > > schema
> > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > property of the
> > > > > > > > transport and not the stream format itself. If we are leaning
> > > > towards just
> > > > > > > > schema evolution then maybe it makes sense to discuss it for the
> > > > IPC stream
> > > > > > > > format and leverage that in Flight. I'd be interested in what
> > > > others think.
> > > > > > >
> > > > > > > I tend to agree, I think stream multiplexing is likely a transport
> > > > level
> > > > > > > issue.  IMO I think schema evolution should be consistent with the
> > > > IPC
> > > > > > > stream format  and flight.
> > > > > > >
> > > > > > >
> > > > > > > > Nate: it may be worth starting a separate discussion about more
> > > > general
> > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > metadata was
> > > > > > > > chosen/if opaque bytes were considered in the past.
> > > > > > >
> > > > > > >
> > > > > > > I think  this was an unfortunate design of the key value metadata
> > > in
> > > > > > > Schema.fbs, but I don't think I was around when this decision was
> > > > made.
> > > > > >
> > > > > > I agree that it's unfortunate that we did not use [ byte ] instead of
> > > > > > string for the value in the KeyValue metadata — I think this was more
> > > > > > of an oversight than a deliberate choice (e.g. it was not our intent
> > > > > > to require binary data to be base64-encoded — this is something that
> > > > > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > > > > for Parquet, for example). Is the binary representation of [byte]
> > > > > > different from string?
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Side Question: Why isn't the IPC stream format a series of the
> > > flight
> > > > > > > > protobufs?
> > > > > > >
> > > > > > > In addition to what David said, protobufs can't be read directly
> > > > from a
> > > > > > > memory-mapped file (they need decoding).  This was one of the
> > > design
> > > > > > > considerations of using flatbuffers and IPC Stream/File format.
> > > > > > >
> > > > > > > I was thinking Micah's comment is more that whatever we do, it
> > > > should be
> > > > > > > > clearly specified and edge cases should be considered, especially
> > > > if we
> > > > > > > > might want to 'backport' this into the stream format later.
> > > > > > >
> > > > > > >
> > > > > > > Yes, for dictionaries we just need to be careful to define
> > > semantics
> > > > and
> > > > > > > ensure implementations are validating them with regards to
> > > > dictionaries.
> > > > > > > There likely isn't any need to change current implementations
> > > though.
> > > > > > >
> > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > > > schema
> > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > property of the
> > > > > > > > transport and not the stream format itself. If we are leaning
> > > > towards just
> > > > > > > > schema evolution then maybe it makes sense to discuss it for the
> > > > IPC stream
> > > > > > > > format and leverage that in Flight. I'd be interested in what
> > > > others think.
> > > > > > > >
> > > > > > > > Especially if we are looking at multiplexing streams - I would
> > > > wonder if
> > > > > > > > that's actually better served by making it easier to implement
> > > > using the
> > > > > > > > Flight implementation as it stands (by managing concurrent RPC
> > > > calls and/or
> > > > > > > > performing the union-of-structs encoding trick for you), instead
> > > > of having
> > > > > > > > to change the protocol.
> > > > > > > >
> > > > > > > > Nate: it may be worth starting a separate discussion about more
> > > > general
> > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > metadata was
> > > > > > > > chosen/if opaque bytes were considered in the past. Off the top
> > > of
> > > > my head
> > > > > > > > if it's for on-disk storage and fully application-defined it may
> > > > make sense
> > > > > > > > to store as a separate file alongside the Arrow file (indexed by
> > > > record
> > > > > > > > batch index) where you can take advantage of whatever format is
> > > > most
> > > > > > > > suitable.
> > > > > > > >
> > > > > > > > -David
> > > > > > > >
> > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> > > to
> > > > add this
> > > > > > > > > feature starting from the IPC(I moved initial write up steps to
> > > > the
> > > > > > > > bottom
> > > > > > > > > of the doc). Afterwards David suggested focusing on Flight and
> > > > that's how
> > > > > > > > > we ended up with the protobufs change in the proposal. This
> > > > being said I
> > > > > > > > do
> > > > > > > > > think that the place where this should be impemented is a good
> > > > question
> > > > > > > > on
> > > > > > > > > its own. Maybe it makes sense to have this kind of a feature in
> > > > IPC and
> > > > > > > > > somehow use it in Flight, maybe not.
> > > > > > > > > 2. The point about dictionaries deserves a dedicated section in
> > > > the
> > > > > > > > > proposal. Nate and David brought it up and shared some
> > > insights.
> > > > I'll try
> > > > > > > > > to aggregate them and we can continue the discussion form
> > > there.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Gosh
> > > > > > > > >
> > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > > > natebauernfeind@deephaven.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > makes it more difficult to bring schema evolution back
> > > > into the
> > > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > > > > >
> > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
> > > > > > > > protobufs.
> > > > > > > > > > Can
> > > > > > > > > > > > you help me understand how difficult it would be to bring
> > > > the
> > > > > > > > > > `schema_id`
> > > > > > > > > > > > approach to the IPC stream format?
> > > > > > > > > > >
> > > > > > > > > > > I thought we were talking solely about the Flight Protobuf
> > > > > > > > definitions -
> > > > > > > > > > > not the Flatbuffers (and the Google doc at least only talks
> > > > about the
> > > > > > > > > > > Protobufs).
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I somehow missed that schema_id is being added to protobuf in
> > > > the
> > > > > > > > document.
> > > > > > > > > > It feels to me that the schema_id is a property that would
> > > > ideally only
> > > > > > > > > > apply to the RecordBatch. I better understand Micah's
> > > > dictionary
> > > > > > > > concerns,
> > > > > > > > > > now, too.
> > > > > > > > > >
> > > > > > > > > > > Side Question: Why isn't the IPC stream format a series of
> > > > the flight
> > > > > > > > > > > > protobufs? It's a real shame that there is no standard
> > > way
> > > > to
> > > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> > > > ignoring the
> > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > > > >
> > > > > > > > > > > The IPC format was defined long before Flight, and Flight's
> > > > > > > > app_metadata
> > > > > > > > > > > was added after Flight's initial definition. Note an IPC
> > > > message does
> > > > > > > > > > have
> > > > > > > > > > > a provision for key-value metadata, though I think APIs for
> > > > that are
> > > > > > > > not
> > > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > > despite my
> > > > > > > > comments
> > > > > > > > > > > there perhaps we need to unify or at least consider how
> > > > Flight's
> > > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> > > Also
> > > > > > > > perhaps see
> > > > > > > > > > > ARROW-1059.)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> > > > strings are
> > > > > > > > only
> > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> > > > opaque
> > > > > > > > bytes.
> > > > > > > > > > The latter is a bit more useful.
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> > >
> > > --
> > >

Re: [DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

The cost of an empty vector in Flatbuffers appears to be 4 bytes.

On Wed, Jul 7, 2021 at 5:50 PM Micah Kornfield <em...@gmail.com> wrote:
>
> Retitling and forking the discussion to talk about key value pairs.
>
> What is the byte cost of an empty list?  Another option would be to
> introduce a new BinaryKeyValue table and add binary metadata.
>
> On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> natebauernfeind@deephaven.io> wrote:
>
> > Deephaven and I are very supportive of "upgrading" the value half of the kv
> > pair to a byte vector. What is the best way to find out if there is
> > sufficient interest?
> >
> >
> > I've been stewing on the ideas here around schema evolution, and I realize
> > the specific feature I am missing is the ability to encode that a field
> > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > empty/has-no-data in O(0) cost (yes; for free).
> >
> > Might there be interest in adding a "field_id" to the FieldNode (which is
> > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > upgrade (by either keying off of 0, or explicitly set the field default to
> > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > length of zero, and 2) all Buffer's associated at that level (and further
> > nested) are also equally empty (i.e. Buffer length is zero).
> >
> > I understand this concept slightly interferes with RecordBatch's `length`
> > field, and that many implementations use that length to resize the
> > root-level FieldNodes. The use-case I have in mind has different logical
> > lengths per field node; current implementations require sending a
> > RecordBatch length of the max length across all root level field nodes. I
> > believe this requires a copy of data whenever a field node is too short; I
> > don't know if there is a decent solution to this slight inefficiency. I am
> > bringing it up because if "skipping a field node when it is empty" is a
> > feature, then we may not want to allocate space for those nodes given that
> > the record batch length will likely be greater than zero.
> >
> > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> > > >
> > > > From the Flatbuffers internals doc[1] it appears they are the same:
> > > "Strings are simply a vector of bytes, and are always null-terminated."
> > >
> > > I see. I took a look at flatbuffers.h, and it appears that changing
> > > this field from string to [byte] would be backward-compatible and
> > > forward-compatible except with code that expects a null terminator.
> > > This is something we could discuss separately if there were enough
> > > interest.
> > >
> > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > >
> > > > -David
> > > >
> > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > > schema
> > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > property of the
> > > > > > > transport and not the stream format itself. If we are leaning
> > > towards just
> > > > > > > schema evolution then maybe it makes sense to discuss it for the
> > > IPC stream
> > > > > > > format and leverage that in Flight. I'd be interested in what
> > > others think.
> > > > > >
> > > > > > I tend to agree, I think stream multiplexing is likely a transport
> > > level
> > > > > > issue.  IMO I think schema evolution should be consistent with the
> > > IPC
> > > > > > stream format  and flight.
> > > > > >
> > > > > >
> > > > > > > Nate: it may be worth starting a separate discussion about more
> > > general
> > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > metadata was
> > > > > > > chosen/if opaque bytes were considered in the past.
> > > > > >
> > > > > >
> > > > > > I think  this was an unfortunate design of the key value metadata
> > in
> > > > > > Schema.fbs, but I don't think I was around when this decision was
> > > made.
> > > > >
> > > > > I agree that it's unfortunate that we did not use [ byte ] instead of
> > > > > string for the value in the KeyValue metadata — I think this was more
> > > > > of an oversight than a deliberate choice (e.g. it was not our intent
> > > > > to require binary data to be base64-encoded — this is something that
> > > > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > > > for Parquet, for example). Is the binary representation of [byte]
> > > > > different from string?
> > > > >
> > > > >
> > > > >
> > > > > > Side Question: Why isn't the IPC stream format a series of the
> > flight
> > > > > > > protobufs?
> > > > > >
> > > > > > In addition to what David said, protobufs can't be read directly
> > > from a
> > > > > > memory-mapped file (they need decoding).  This was one of the
> > design
> > > > > > considerations of using flatbuffers and IPC Stream/File format.
> > > > > >
> > > > > > I was thinking Micah's comment is more that whatever we do, it
> > > should be
> > > > > > > clearly specified and edge cases should be considered, especially
> > > if we
> > > > > > > might want to 'backport' this into the stream format later.
> > > > > >
> > > > > >
> > > > > > Yes, for dictionaries we just need to be careful to define
> > semantics
> > > and
> > > > > > ensure implementations are validating them with regards to
> > > dictionaries.
> > > > > > There likely isn't any need to change current implementations
> > though.
> > > > > >
> > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > > schema
> > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > property of the
> > > > > > > transport and not the stream format itself. If we are leaning
> > > towards just
> > > > > > > schema evolution then maybe it makes sense to discuss it for the
> > > IPC stream
> > > > > > > format and leverage that in Flight. I'd be interested in what
> > > others think.
> > > > > > >
> > > > > > > Especially if we are looking at multiplexing streams - I would
> > > wonder if
> > > > > > > that's actually better served by making it easier to implement
> > > using the
> > > > > > > Flight implementation as it stands (by managing concurrent RPC
> > > calls and/or
> > > > > > > performing the union-of-structs encoding trick for you), instead
> > > of having
> > > > > > > to change the protocol.
> > > > > > >
> > > > > > > Nate: it may be worth starting a separate discussion about more
> > > general
> > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > metadata was
> > > > > > > chosen/if opaque bytes were considered in the past. Off the top
> > of
> > > my head
> > > > > > > if it's for on-disk storage and fully application-defined it may
> > > make sense
> > > > > > > to store as a separate file alongside the Arrow file (indexed by
> > > record
> > > > > > > batch index) where you can take advantage of whatever format is
> > > most
> > > > > > > suitable.
> > > > > > >
> > > > > > > -David
> > > > > > >
> > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> > to
> > > add this
> > > > > > > > feature starting from the IPC(I moved initial write up steps to
> > > the
> > > > > > > bottom
> > > > > > > > of the doc). Afterwards David suggested focusing on Flight and
> > > that's how
> > > > > > > > we ended up with the protobufs change in the proposal. This
> > > being said I
> > > > > > > do
> > > > > > > > think that the place where this should be impemented is a good
> > > question
> > > > > > > on
> > > > > > > > its own. Maybe it makes sense to have this kind of a feature in
> > > IPC and
> > > > > > > > somehow use it in Flight, maybe not.
> > > > > > > > 2. The point about dictionaries deserves a dedicated section in
> > > the
> > > > > > > > proposal. Nate and David brought it up and shared some
> > insights.
> > > I'll try
> > > > > > > > to aggregate them and we can continue the discussion form
> > there.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Gosh
> > > > > > > >
> > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > > natebauernfeind@deephaven.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > makes it more difficult to bring schema evolution back
> > > into the
> > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > > > >
> > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
> > > > > > > protobufs.
> > > > > > > > > Can
> > > > > > > > > > > you help me understand how difficult it would be to bring
> > > the
> > > > > > > > > `schema_id`
> > > > > > > > > > > approach to the IPC stream format?
> > > > > > > > > >
> > > > > > > > > > I thought we were talking solely about the Flight Protobuf
> > > > > > > definitions -
> > > > > > > > > > not the Flatbuffers (and the Google doc at least only talks
> > > about the
> > > > > > > > > > Protobufs).
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I somehow missed that schema_id is being added to protobuf in
> > > the
> > > > > > > document.
> > > > > > > > > It feels to me that the schema_id is a property that would
> > > ideally only
> > > > > > > > > apply to the RecordBatch. I better understand Micah's
> > > dictionary
> > > > > > > concerns,
> > > > > > > > > now, too.
> > > > > > > > >
> > > > > > > > > > Side Question: Why isn't the IPC stream format a series of
> > > the flight
> > > > > > > > > > > protobufs? It's a real shame that there is no standard
> > way
> > > to
> > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> > > ignoring the
> > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > > >
> > > > > > > > > > The IPC format was defined long before Flight, and Flight's
> > > > > > > app_metadata
> > > > > > > > > > was added after Flight's initial definition. Note an IPC
> > > message does
> > > > > > > > > have
> > > > > > > > > > a provision for key-value metadata, though I think APIs for
> > > that are
> > > > > > > not
> > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > despite my
> > > > > > > comments
> > > > > > > > > > there perhaps we need to unify or at least consider how
> > > Flight's
> > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> > Also
> > > > > > > perhaps see
> > > > > > > > > > ARROW-1059.)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> > > strings are
> > > > > > > only
> > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> > > opaque
> > > > > > > bytes.
> > > > > > > > > The latter is a bit more useful.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> >
> > --
> >

[DISCUSS] Binary Values in Key value pairs WAS: Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

Retitling and forking the discussion to talk about key value pairs.

What is the byte cost of an empty list?  Another option would be to
introduce a new BinaryKeyValue table and add binary metadata.

On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
natebauernfeind@deephaven.io> wrote:

> Deephaven and I are very supportive of "upgrading" the value half of the kv
> pair to a byte vector. What is the best way to find out if there is
> sufficient interest?
>
>
> I've been stewing on the ideas here around schema evolution, and I realize
> the specific feature I am missing is the ability to encode that a field
> (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> empty/has-no-data in O(0) cost (yes; for free).
>
> Might there be interest in adding a "field_id" to the FieldNode (which is
> encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> upgrade (by either keying off of 0, or explicitly set the field default to
> -1) which would allow the sender to "skip" fields that have 1) FieldNode
> length of zero, and 2) all Buffer's associated at that level (and further
> nested) are also equally empty (i.e. Buffer length is zero).
>
> I understand this concept slightly interferes with RecordBatch's `length`
> field, and that many implementations use that length to resize the
> root-level FieldNodes. The use-case I have in mind has different logical
> lengths per field node; current implementations require sending a
> RecordBatch length of the max length across all root level field nodes. I
> believe this requires a copy of data whenever a field node is too short; I
> don't know if there is a decent solution to this slight inefficiency. I am
> bringing it up because if "skipping a field node when it is empty" is a
> feature, then we may not want to allocate space for those nodes given that
> the record batch length will likely be greater than zero.
>
> On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
>
> > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> > >
> > > From the Flatbuffers internals doc[1] it appears they are the same:
> > "Strings are simply a vector of bytes, and are always null-terminated."
> >
> > I see. I took a look at flatbuffers.h, and it appears that changing
> > this field from string to [byte] would be backward-compatible and
> > forward-compatible except with code that expects a null terminator.
> > This is something we could discuss separately if there were enough
> > interest.
> >
> > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> > >
> > > -David
> > >
> > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> emkornfield@gmail.com>
> > wrote:
> > > > >
> > > > > >
> > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > schema
> > > > > > evolution or multiplexing streams (more so the latter) is a
> > property of the
> > > > > > transport and not the stream format itself. If we are leaning
> > towards just
> > > > > > schema evolution then maybe it makes sense to discuss it for the
> > IPC stream
> > > > > > format and leverage that in Flight. I'd be interested in what
> > others think.
> > > > >
> > > > > I tend to agree, I think stream multiplexing is likely a transport
> > level
> > > > > issue.  IMO I think schema evolution should be consistent with the
> > IPC
> > > > > stream format  and flight.
> > > > >
> > > > >
> > > > > > Nate: it may be worth starting a separate discussion about more
> > general
> > > > > > metadata in the IPC message. I'm not aware of why key-value
> > metadata was
> > > > > > chosen/if opaque bytes were considered in the past.
> > > > >
> > > > >
> > > > > I think  this was an unfortunate design of the key value metadata
> in
> > > > > Schema.fbs, but I don't think I was around when this decision was
> > made.
> > > >
> > > > I agree that it's unfortunate that we did not use [ byte ] instead of
> > > > string for the value in the KeyValue metadata — I think this was more
> > > > of an oversight than a deliberate choice (e.g. it was not our intent
> > > > to require binary data to be base64-encoded — this is something that
> > > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > > for Parquet, for example). Is the binary representation of [byte]
> > > > different from string?
> > > >
> > > >
> > > >
> > > > > Side Question: Why isn't the IPC stream format a series of the
> flight
> > > > > > protobufs?
> > > > >
> > > > > In addition to what David said, protobufs can't be read directly
> > from a
> > > > > memory-mapped file (they need decoding).  This was one of the
> design
> > > > > considerations of using flatbuffers and IPC Stream/File format.
> > > > >
> > > > > I was thinking Micah's comment is more that whatever we do, it
> > should be
> > > > > > clearly specified and edge cases should be considered, especially
> > if we
> > > > > > might want to 'backport' this into the stream format later.
> > > > >
> > > > >
> > > > > Yes, for dictionaries we just need to be careful to define
> semantics
> > and
> > > > > ensure implementations are validating them with regards to
> > dictionaries.
> > > > > There likely isn't any need to change current implementations
> though.
> > > > >
> > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> > wrote:
> > > > >
> > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > schema
> > > > > > evolution or multiplexing streams (more so the latter) is a
> > property of the
> > > > > > transport and not the stream format itself. If we are leaning
> > towards just
> > > > > > schema evolution then maybe it makes sense to discuss it for the
> > IPC stream
> > > > > > format and leverage that in Flight. I'd be interested in what
> > others think.
> > > > > >
> > > > > > Especially if we are looking at multiplexing streams - I would
> > wonder if
> > > > > > that's actually better served by making it easier to implement
> > using the
> > > > > > Flight implementation as it stands (by managing concurrent RPC
> > calls and/or
> > > > > > performing the union-of-structs encoding trick for you), instead
> > of having
> > > > > > to change the protocol.
> > > > > >
> > > > > > Nate: it may be worth starting a separate discussion about more
> > general
> > > > > > metadata in the IPC message. I'm not aware of why key-value
> > metadata was
> > > > > > chosen/if opaque bytes were considered in the past. Off the top
> of
> > my head
> > > > > > if it's for on-disk storage and fully application-defined it may
> > make sense
> > > > > > to store as a separate file alongside the Arrow file (indexed by
> > record
> > > > > > batch index) where you can take advantage of whatever format is
> > most
> > > > > > suitable.
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> to
> > add this
> > > > > > > feature starting from the IPC(I moved initial write up steps to
> > the
> > > > > > bottom
> > > > > > > of the doc). Afterwards David suggested focusing on Flight and
> > that's how
> > > > > > > we ended up with the protobufs change in the proposal. This
> > being said I
> > > > > > do
> > > > > > > think that the place where this should be impemented is a good
> > question
> > > > > > on
> > > > > > > its own. Maybe it makes sense to have this kind of a feature in
> > IPC and
> > > > > > > somehow use it in Flight, maybe not.
> > > > > > > 2. The point about dictionaries deserves a dedicated section in
> > the
> > > > > > > proposal. Nate and David brought it up and shared some
> insights.
> > I'll try
> > > > > > > to aggregate them and we can continue the discussion form
> there.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Gosh
> > > > > > >
> > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > natebauernfeind@deephaven.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > >
> > > > > > > > > > > makes it more difficult to bring schema evolution back
> > into the
> > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > > >
> > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
> > > > > > protobufs.
> > > > > > > > Can
> > > > > > > > > > you help me understand how difficult it would be to bring
> > the
> > > > > > > > `schema_id`
> > > > > > > > > > approach to the IPC stream format?
> > > > > > > > >
> > > > > > > > > I thought we were talking solely about the Flight Protobuf
> > > > > > definitions -
> > > > > > > > > not the Flatbuffers (and the Google doc at least only talks
> > about the
> > > > > > > > > Protobufs).
> > > > > > > > >
> > > > > > > >
> > > > > > > > I somehow missed that schema_id is being added to protobuf in
> > the
> > > > > > document.
> > > > > > > > It feels to me that the schema_id is a property that would
> > ideally only
> > > > > > > > apply to the RecordBatch. I better understand Micah's
> > dictionary
> > > > > > concerns,
> > > > > > > > now, too.
> > > > > > > >
> > > > > > > > > Side Question: Why isn't the IPC stream format a series of
> > the flight
> > > > > > > > > > protobufs? It's a real shame that there is no standard
> way
> > to
> > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> > ignoring the
> > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > >
> > > > > > > > > The IPC format was defined long before Flight, and Flight's
> > > > > > app_metadata
> > > > > > > > > was added after Flight's initial definition. Note an IPC
> > message does
> > > > > > > > have
> > > > > > > > > a provision for key-value metadata, though I think APIs for
> > that are
> > > > > > not
> > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > despite my
> > > > > > comments
> > > > > > > > > there perhaps we need to unify or at least consider how
> > Flight's
> > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> Also
> > > > > > perhaps see
> > > > > > > > > ARROW-1059.)
> > > > > > > > >
> > > > > > > >
> > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> > strings are
> > > > > > only
> > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> > opaque
> > > > > > bytes.
> > > > > > > > The latter is a bit more useful.
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>
>
> --
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

>
> It sounds like we may want to discuss some potential evolutions of the
> Arrow binary protocol (for example: new Message types).


This sounds like a good path forward.

On Fri, Jul 9, 2021 at 8:26 AM Wes McKinney <we...@gmail.com> wrote:

> It sounds like we may want to discuss some potential evolutions of the
> Arrow binary protocol (for example: new Message types). Certainly a
> can of worms but rather than trying to bolt some new functionality
> onto the existing structures, it might be better to support the new
> use cases through some new structures which will be more clear cut
> from a forward compatibility standpoint.
>
> On Wed, Jul 7, 2021 at 8:31 PM David Li <li...@apache.org> wrote:
> >
> > To summarize so far, it sounds like schema evolution is neither
> sufficient nor necessary for either Gosh or Nate's use-cases here? It could
> be useful for FlightSQL but even there I don't think it's a requirement.
> >
> > For Nate - it almost sounds like what you need is some way to slice up a
> record batch and send columns individually, which isn't really a concept in
> IPC (and hence Flight). Or rather, record batch is almost the wrong
> abstraction for your use case (when you're sending per-column deltas), even
> though you could model it as a record batch with 'empty' columns or as a
> stream with constantly shifting schema (neither of which are perfect
> encodings).
> >
> > -David
> >
> > On Wed, Jul 7, 2021, at 13:24, Nate Bauernfeind wrote:
> > > > Flatbuffers does not support modifying structs
> > > > in any forwards or backwards compatible way
> > > > (only tables support evolution).
> > >
> > > Bah. I did not realize that.
> > >
> > > To reiterate the feature that would be ideal:
> > > I realize the specific feature I am missing is the ability to encode
> that a
> > > field (i.e. its FieldNode and accompanying Buffers in the RecordBatch)
> is
> > > empty/has-no-data in O(0) cost (yes; for free).
> > >
> > > Since RecordBatch is a table, might there be interest in adding a field
> > > that is a bitset (formatted as a byte vector) to indicate the subset of
> > > root FieldNodes that are included in the RecordBatch's field-node-list
> > > (noting that the buffer list most be appropriately filtered, too)? If
> the
> > > bitset is omitted/empty we can be forwards-compatible by assuming every
> > > field is included (seems fair; because why send a record batch without
> a
> > > payload?). This does bring down the cost from 16 bytes (field node) +
> 32
> > > bytes (2x buffer node minimum; validity buffer node + payload buffer
> node)
> > > to 1 bit per field node -- that's a compression ratio of >= 384:1 --
> > > although it's not free, it's a lot closer.
> > >
> > > Are there any other alternatives the Arrow community might consider?
> > >
> > >
> > > > This sounds a lot like what DenseUnion provides though?
> > >
> > > The use case is as follows: We send multiple record batches that
> aggregate
> > > / accumulate into a single logical update. The first set of record
> batches
> > > contain payload for rows that were added since the last logical update
> > > (this is a set of updates to accommodate that not every implementation
> > > supports 32-bit lengths). A field node for every column and every
> added row
> > > will be sent. For this half of the logical update the RecordBatch
> length
> > > matches the root FieldNode lengths. The second set of record batches
> > > contain payload for only rows, and fields, that were actually
> modified. If
> > > only one field changed, we send only that payload for that row. There
> is
> > > additional metadata that allows the client to understand which
> existing row
> > > is to be replaced by any given row for a given Field/Column. In this
> > > context, each root field node has a different mapping to minimize total
> > > payload size.
> > >
> > > We might be able to use DenseUnion but it certainly feels like folding
> the
> > > entire data-source into a dense union makes the design less useful and
> less
> > > accessible. I will spend some time sleeping on your suggestion, but
> I'm not
> > > immediately excited about it. At this moment, I suspect I will
> continue to
> > > lie and state that the RecordBatch's length is the max length across
> all
> > > root field node lengths (and be content that it's not ideal from a
> > > copy/allocation perspective).
> > > -
> > >
> > > On Wed, Jul 7, 2021 at 10:57 AM Micah Kornfield <emkornfield@gmail.com
> >
> > > wrote:
> > >
> > > > >
> > > > > Might there be interest in adding a "field_id" to the FieldNode
> (which is
> > > > > encoded on the RecordBatch flatbuffer)? I see a simple
> forward-compatible
> > > > > upgrade (by either keying off of 0, or explicitly set the field
> default
> > > > to
> > > > > -1) which would allow the sender to "skip" fields that have 1)
> FieldNode
> > > > > length of zero, and 2) all Buffer's associated at that level (and
> further
> > > > > nested) are also equally empty (i.e. Buffer length is zero).
> > > >
> > > >
> > > > FieldNode is a struct in Message.fbs.   Flatbuffers does not support
> > > > modifying structs in any forwards or backwards compatible way (only
> tables
> > > > support evolution).  I think there was originally more metadata in
> > > > FieldNode and it was stripped down due to size concerns.
> > > >
> > > > I understand this concept slightly interferes with RecordBatch's
> `length`
> > > > > field, and that many implementations use that length to resize the
> > > > > root-level FieldNodes. The use-case I have in mind has different
> logical
> > > > > lengths per field node; current implementations require sending a
> > > > > RecordBatch length of the max length across all root level field
> nodes. I
> > > > > believe this requires a copy of data whenever a field node is too
> short;
> > > > I
> > > > > don't know if there is a decent solution to this slight
> inefficiency. I
> > > > am
> > > > > bringing it up because if "skipping a field node when it is empty"
> is a
> > > > > feature, then we may not want to allocate space for those nodes
> given
> > > > that
> > > > > the record batch length will likely be greater than zero.
> > > >
> > > >
> > > > Having conflicting RecordBatch and top-level field nodes is
> something that
> > > > I believe we have pushed back on in the past.   This sounds a lot
> like
> > > > what  DenseUnion provides though?
> > > >
> > > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > > > natebauernfeind@deephaven.io> wrote:
> > > >
> > > > > Deephaven and I are very supportive of "upgrading" the value half
> of the
> > > > kv
> > > > > pair to a byte vector. What is the best way to find out if there is
> > > > > sufficient interest?
> > > > >
> > > > >
> > > > > I've been stewing on the ideas here around schema evolution, and I
> > > > realize
> > > > > the specific feature I am missing is the ability to encode that a
> field
> > > > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > > > > empty/has-no-data in O(0) cost (yes; for free).
> > > > >
> > > > > Might there be interest in adding a "field_id" to the FieldNode
> (which is
> > > > > encoded on the RecordBatch flatbuffer)? I see a simple
> forward-compatible
> > > > > upgrade (by either keying off of 0, or explicitly set the field
> default
> > > > to
> > > > > -1) which would allow the sender to "skip" fields that have 1)
> FieldNode
> > > > > length of zero, and 2) all Buffer's associated at that level (and
> further
> > > > > nested) are also equally empty (i.e. Buffer length is zero).
> > > > >
> > > > > I understand this concept slightly interferes with RecordBatch's
> `length`
> > > > > field, and that many implementations use that length to resize the
> > > > > root-level FieldNodes. The use-case I have in mind has different
> logical
> > > > > lengths per field node; current implementations require sending a
> > > > > RecordBatch length of the max length across all root level field
> nodes. I
> > > > > believe this requires a copy of data whenever a field node is too
> short;
> > > > I
> > > > > don't know if there is a decent solution to this slight
> inefficiency. I
> > > > am
> > > > > bringing it up because if "skipping a field node when it is empty"
> is a
> > > > > feature, then we may not want to allocate space for those nodes
> given
> > > > that
> > > > > the record batch length will likely be greater than zero.
> > > > >
> > > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com>
> wrote:
> > > > >
> > > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me>
> wrote:
> > > > > > >
> > > > > > > From the Flatbuffers internals doc[1] it appears they are the
> same:
> > > > > > "Strings are simply a vector of bytes, and are always
> null-terminated."
> > > > > >
> > > > > > I see. I took a look at flatbuffers.h, and it appears that
> changing
> > > > > > this field from string to [byte] would be backward-compatible and
> > > > > > forward-compatible except with code that expects a null
> terminator.
> > > > > > This is something we could discuss separately if there were
> enough
> > > > > > interest.
> > > > > >
> > > > > > > [1]:
> https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > > > > >
> > > > > > > -David
> > > > > > >
> > > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > > > > emkornfield@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Right, I had wanted to focus the discussion on Flight as
> I
> > > > think
> > > > > > schema
> > > > > > > > > > evolution or multiplexing streams (more so the latter)
> is a
> > > > > > property of the
> > > > > > > > > > transport and not the stream format itself. If we are
> leaning
> > > > > > towards just
> > > > > > > > > > schema evolution then maybe it makes sense to discuss it
> for
> > > > the
> > > > > > IPC stream
> > > > > > > > > > format and leverage that in Flight. I'd be interested in
> what
> > > > > > others think.
> > > > > > > > >
> > > > > > > > > I tend to agree, I think stream multiplexing is likely a
> > > > transport
> > > > > > level
> > > > > > > > > issue.  IMO I think schema evolution should be consistent
> with
> > > > the
> > > > > > IPC
> > > > > > > > > stream format  and flight.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Nate: it may be worth starting a separate discussion
> about more
> > > > > > general
> > > > > > > > > > metadata in the IPC message. I'm not aware of why
> key-value
> > > > > > metadata was
> > > > > > > > > > chosen/if opaque bytes were considered in the past.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I think  this was an unfortunate design of the key value
> metadata
> > > > > in
> > > > > > > > > Schema.fbs, but I don't think I was around when this
> decision was
> > > > > > made.
> > > > > > > >
> > > > > > > > I agree that it's unfortunate that we did not use [ byte ]
> instead
> > > > of
> > > > > > > > string for the value in the KeyValue metadata — I think this
> was
> > > > more
> > > > > > > > of an oversight than a deliberate choice (e.g. it was not our
> > > > intent
> > > > > > > > to require binary data to be base64-encoded — this is
> something
> > > > that
> > > > > > > > we have to do when encoding binary data in Thrift KeyValue
> metadata
> > > > > > > > for Parquet, for example). Is the binary representation of
> [byte]
> > > > > > > > different from string?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > Side Question: Why isn't the IPC stream format a series of
> the
> > > > > flight
> > > > > > > > > > protobufs?
> > > > > > > > >
> > > > > > > > > In addition to what David said, protobufs can't be read
> directly
> > > > > > from a
> > > > > > > > > memory-mapped file (they need decoding).  This was one of
> the
> > > > > design
> > > > > > > > > considerations of using flatbuffers and IPC Stream/File
> format.
> > > > > > > > >
> > > > > > > > > I was thinking Micah's comment is more that whatever we
> do, it
> > > > > > should be
> > > > > > > > > > clearly specified and edge cases should be considered,
> > > > especially
> > > > > > if we
> > > > > > > > > > might want to 'backport' this into the stream format
> later.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yes, for dictionaries we just need to be careful to define
> > > > > semantics
> > > > > > and
> > > > > > > > > ensure implementations are validating them with regards to
> > > > > > dictionaries.
> > > > > > > > > There likely isn't any need to change current
> implementations
> > > > > though.
> > > > > > > > >
> > > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <
> lidavidm@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Right, I had wanted to focus the discussion on Flight as
> I
> > > > think
> > > > > > schema
> > > > > > > > > > evolution or multiplexing streams (more so the latter)
> is a
> > > > > > property of the
> > > > > > > > > > transport and not the stream format itself. If we are
> leaning
> > > > > > towards just
> > > > > > > > > > schema evolution then maybe it makes sense to discuss it
> for
> > > > the
> > > > > > IPC stream
> > > > > > > > > > format and leverage that in Flight. I'd be interested in
> what
> > > > > > others think.
> > > > > > > > > >
> > > > > > > > > > Especially if we are looking at multiplexing streams - I
> would
> > > > > > wonder if
> > > > > > > > > > that's actually better served by making it easier to
> implement
> > > > > > using the
> > > > > > > > > > Flight implementation as it stands (by managing
> concurrent RPC
> > > > > > calls and/or
> > > > > > > > > > performing the union-of-structs encoding trick for you),
> > > > instead
> > > > > > of having
> > > > > > > > > > to change the protocol.
> > > > > > > > > >
> > > > > > > > > > Nate: it may be worth starting a separate discussion
> about more
> > > > > > general
> > > > > > > > > > metadata in the IPC message. I'm not aware of why
> key-value
> > > > > > metadata was
> > > > > > > > > > chosen/if opaque bytes were considered in the past. Off
> the top
> > > > > of
> > > > > > my head
> > > > > > > > > > if it's for on-disk storage and fully
> application-defined it
> > > > may
> > > > > > make sense
> > > > > > > > > > to store as a separate file alongside the Arrow file
> (indexed
> > > > by
> > > > > > record
> > > > > > > > > > batch index) where you can take advantage of whatever
> format is
> > > > > > most
> > > > > > > > > > suitable.
> > > > > > > > > >
> > > > > > > > > > -David
> > > > > > > > > >
> > > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > > > > > Hi guys,
> > > > > > > > > > >
> > > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial
> suggestion was
> > > > > to
> > > > > > add this
> > > > > > > > > > > feature starting from the IPC(I moved initial write up
> steps
> > > > to
> > > > > > the
> > > > > > > > > > bottom
> > > > > > > > > > > of the doc). Afterwards David suggested focusing on
> Flight
> > > > and
> > > > > > that's how
> > > > > > > > > > > we ended up with the protobufs change in the proposal.
> This
> > > > > > being said I
> > > > > > > > > > do
> > > > > > > > > > > think that the place where this should be impemented
> is a
> > > > good
> > > > > > question
> > > > > > > > > > on
> > > > > > > > > > > its own. Maybe it makes sense to have this kind of a
> feature
> > > > in
> > > > > > IPC and
> > > > > > > > > > > somehow use it in Flight, maybe not.
> > > > > > > > > > > 2. The point about dictionaries deserves a dedicated
> section
> > > > in
> > > > > > the
> > > > > > > > > > > proposal. Nate and David brought it up and shared some
> > > > > insights.
> > > > > > I'll try
> > > > > > > > > > > to aggregate them and we can continue the discussion
> form
> > > > > there.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Gosh
> > > > > > > > > > >
> > > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > > > > > natebauernfeind@deephaven.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > makes it more difficult to bring schema
> evolution
> > > > back
> > > > > > into the
> > > > > > > > > > > > > > > IPC Stream format (i.e. it would live only in
> flight)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Gosh's proposal extends the flatbuffer
> structures not
> > > > the
> > > > > > > > > > protobufs.
> > > > > > > > > > > > Can
> > > > > > > > > > > > > > you help me understand how difficult it would be
> to
> > > > bring
> > > > > > the
> > > > > > > > > > > > `schema_id`
> > > > > > > > > > > > > > approach to the IPC stream format?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I thought we were talking solely about the Flight
> > > > Protobuf
> > > > > > > > > > definitions -
> > > > > > > > > > > > > not the Flatbuffers (and the Google doc at least
> only
> > > > talks
> > > > > > about the
> > > > > > > > > > > > > Protobufs).
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I somehow missed that schema_id is being added to
> protobuf
> > > > in
> > > > > > the
> > > > > > > > > > document.
> > > > > > > > > > > > It feels to me that the schema_id is a property that
> would
> > > > > > ideally only
> > > > > > > > > > > > apply to the RecordBatch. I better understand Micah's
> > > > > > dictionary
> > > > > > > > > > concerns,
> > > > > > > > > > > > now, too.
> > > > > > > > > > > >
> > > > > > > > > > > > > Side Question: Why isn't the IPC stream format a
> series
> > > > of
> > > > > > the flight
> > > > > > > > > > > > > > protobufs? It's a real shame that there is no
> standard
> > > > > way
> > > > > > to
> > > > > > > > > > > > > > capture/replay a stream with app_metadata.
> (Obviously
> > > > > > ignoring the
> > > > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > The IPC format was defined long before Flight, and
> > > > Flight's
> > > > > > > > > > app_metadata
> > > > > > > > > > > > > was added after Flight's initial definition. Note
> an IPC
> > > > > > message does
> > > > > > > > > > > > have
> > > > > > > > > > > > > a provision for key-value metadata, though I think
> APIs
> > > > for
> > > > > > that are
> > > > > > > > > > not
> > > > > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940
> and
> > > > > > despite my
> > > > > > > > > > comments
> > > > > > > > > > > > > there perhaps we need to unify or at least
> consider how
> > > > > > Flight's
> > > > > > > > > > > > > app_metadata relates to the IPC message
> custom_metadata.
> > > > > Also
> > > > > > > > > > perhaps see
> > > > > > > > > > > > > ARROW-1059.)
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > KeyValue unfortunately is string to string. In
> flatbuffer
> > > > > > strings are
> > > > > > > > > > only
> > > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other
> hand is
> > > > > > opaque
> > > > > > > > > > bytes.
> > > > > > > > > > > > The latter is a bit more useful.
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > >
> > >
> > >
> > > --
> > >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

It sounds like we may want to discuss some potential evolutions of the
Arrow binary protocol (for example: new Message types). Certainly a
can of worms but rather than trying to bolt some new functionality
onto the existing structures, it might be better to support the new
use cases through some new structures which will be more clear cut
from a forward compatibility standpoint.

On Wed, Jul 7, 2021 at 8:31 PM David Li <li...@apache.org> wrote:
>
> To summarize so far, it sounds like schema evolution is neither sufficient nor necessary for either Gosh or Nate's use-cases here? It could be useful for FlightSQL but even there I don't think it's a requirement.
>
> For Nate - it almost sounds like what you need is some way to slice up a record batch and send columns individually, which isn't really a concept in IPC (and hence Flight). Or rather, record batch is almost the wrong abstraction for your use case (when you're sending per-column deltas), even though you could model it as a record batch with 'empty' columns or as a stream with constantly shifting schema (neither of which are perfect encodings).
>
> -David
>
> On Wed, Jul 7, 2021, at 13:24, Nate Bauernfeind wrote:
> > > Flatbuffers does not support modifying structs
> > > in any forwards or backwards compatible way
> > > (only tables support evolution).
> >
> > Bah. I did not realize that.
> >
> > To reiterate the feature that would be ideal:
> > I realize the specific feature I am missing is the ability to encode that a
> > field (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > empty/has-no-data in O(0) cost (yes; for free).
> >
> > Since RecordBatch is a table, might there be interest in adding a field
> > that is a bitset (formatted as a byte vector) to indicate the subset of
> > root FieldNodes that are included in the RecordBatch's field-node-list
> > (noting that the buffer list most be appropriately filtered, too)? If the
> > bitset is omitted/empty we can be forwards-compatible by assuming every
> > field is included (seems fair; because why send a record batch without a
> > payload?). This does bring down the cost from 16 bytes (field node) + 32
> > bytes (2x buffer node minimum; validity buffer node + payload buffer node)
> > to 1 bit per field node -- that's a compression ratio of >= 384:1 --
> > although it's not free, it's a lot closer.
> >
> > Are there any other alternatives the Arrow community might consider?
> >
> >
> > > This sounds a lot like what DenseUnion provides though?
> >
> > The use case is as follows: We send multiple record batches that aggregate
> > / accumulate into a single logical update. The first set of record batches
> > contain payload for rows that were added since the last logical update
> > (this is a set of updates to accommodate that not every implementation
> > supports 32-bit lengths). A field node for every column and every added row
> > will be sent. For this half of the logical update the RecordBatch length
> > matches the root FieldNode lengths. The second set of record batches
> > contain payload for only rows, and fields, that were actually modified. If
> > only one field changed, we send only that payload for that row. There is
> > additional metadata that allows the client to understand which existing row
> > is to be replaced by any given row for a given Field/Column. In this
> > context, each root field node has a different mapping to minimize total
> > payload size.
> >
> > We might be able to use DenseUnion but it certainly feels like folding the
> > entire data-source into a dense union makes the design less useful and less
> > accessible. I will spend some time sleeping on your suggestion, but I'm not
> > immediately excited about it. At this moment, I suspect I will continue to
> > lie and state that the RecordBatch's length is the max length across all
> > root field node lengths (and be content that it's not ideal from a
> > copy/allocation perspective).
> > -
> >
> > On Wed, Jul 7, 2021 at 10:57 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > >
> > > > Might there be interest in adding a "field_id" to the FieldNode (which is
> > > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > > > upgrade (by either keying off of 0, or explicitly set the field default
> > > to
> > > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > > > length of zero, and 2) all Buffer's associated at that level (and further
> > > > nested) are also equally empty (i.e. Buffer length is zero).
> > >
> > >
> > > FieldNode is a struct in Message.fbs.   Flatbuffers does not support
> > > modifying structs in any forwards or backwards compatible way (only tables
> > > support evolution).  I think there was originally more metadata in
> > > FieldNode and it was stripped down due to size concerns.
> > >
> > > I understand this concept slightly interferes with RecordBatch's `length`
> > > > field, and that many implementations use that length to resize the
> > > > root-level FieldNodes. The use-case I have in mind has different logical
> > > > lengths per field node; current implementations require sending a
> > > > RecordBatch length of the max length across all root level field nodes. I
> > > > believe this requires a copy of data whenever a field node is too short;
> > > I
> > > > don't know if there is a decent solution to this slight inefficiency. I
> > > am
> > > > bringing it up because if "skipping a field node when it is empty" is a
> > > > feature, then we may not want to allocate space for those nodes given
> > > that
> > > > the record batch length will likely be greater than zero.
> > >
> > >
> > > Having conflicting RecordBatch and top-level field nodes is something that
> > > I believe we have pushed back on in the past.   This sounds a lot like
> > > what  DenseUnion provides though?
> > >
> > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > > natebauernfeind@deephaven.io> wrote:
> > >
> > > > Deephaven and I are very supportive of "upgrading" the value half of the
> > > kv
> > > > pair to a byte vector. What is the best way to find out if there is
> > > > sufficient interest?
> > > >
> > > >
> > > > I've been stewing on the ideas here around schema evolution, and I
> > > realize
> > > > the specific feature I am missing is the ability to encode that a field
> > > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > > > empty/has-no-data in O(0) cost (yes; for free).
> > > >
> > > > Might there be interest in adding a "field_id" to the FieldNode (which is
> > > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > > > upgrade (by either keying off of 0, or explicitly set the field default
> > > to
> > > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > > > length of zero, and 2) all Buffer's associated at that level (and further
> > > > nested) are also equally empty (i.e. Buffer length is zero).
> > > >
> > > > I understand this concept slightly interferes with RecordBatch's `length`
> > > > field, and that many implementations use that length to resize the
> > > > root-level FieldNodes. The use-case I have in mind has different logical
> > > > lengths per field node; current implementations require sending a
> > > > RecordBatch length of the max length across all root level field nodes. I
> > > > believe this requires a copy of data whenever a field node is too short;
> > > I
> > > > don't know if there is a decent solution to this slight inefficiency. I
> > > am
> > > > bringing it up because if "skipping a field node when it is empty" is a
> > > > feature, then we may not want to allocate space for those nodes given
> > > that
> > > > the record batch length will likely be greater than zero.
> > > >
> > > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
> > > >
> > > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> > > > > >
> > > > > > From the Flatbuffers internals doc[1] it appears they are the same:
> > > > > "Strings are simply a vector of bytes, and are always null-terminated."
> > > > >
> > > > > I see. I took a look at flatbuffers.h, and it appears that changing
> > > > > this field from string to [byte] would be backward-compatible and
> > > > > forward-compatible except with code that expects a null terminator.
> > > > > This is something we could discuss separately if there were enough
> > > > > interest.
> > > > >
> > > > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > > > emkornfield@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I
> > > think
> > > > > schema
> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > > property of the
> > > > > > > > > transport and not the stream format itself. If we are leaning
> > > > > towards just
> > > > > > > > > schema evolution then maybe it makes sense to discuss it for
> > > the
> > > > > IPC stream
> > > > > > > > > format and leverage that in Flight. I'd be interested in what
> > > > > others think.
> > > > > > > >
> > > > > > > > I tend to agree, I think stream multiplexing is likely a
> > > transport
> > > > > level
> > > > > > > > issue.  IMO I think schema evolution should be consistent with
> > > the
> > > > > IPC
> > > > > > > > stream format  and flight.
> > > > > > > >
> > > > > > > >
> > > > > > > > > Nate: it may be worth starting a separate discussion about more
> > > > > general
> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > > metadata was
> > > > > > > > > chosen/if opaque bytes were considered in the past.
> > > > > > > >
> > > > > > > >
> > > > > > > > I think  this was an unfortunate design of the key value metadata
> > > > in
> > > > > > > > Schema.fbs, but I don't think I was around when this decision was
> > > > > made.
> > > > > > >
> > > > > > > I agree that it's unfortunate that we did not use [ byte ] instead
> > > of
> > > > > > > string for the value in the KeyValue metadata — I think this was
> > > more
> > > > > > > of an oversight than a deliberate choice (e.g. it was not our
> > > intent
> > > > > > > to require binary data to be base64-encoded — this is something
> > > that
> > > > > > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > > > > > for Parquet, for example). Is the binary representation of [byte]
> > > > > > > different from string?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Side Question: Why isn't the IPC stream format a series of the
> > > > flight
> > > > > > > > > protobufs?
> > > > > > > >
> > > > > > > > In addition to what David said, protobufs can't be read directly
> > > > > from a
> > > > > > > > memory-mapped file (they need decoding).  This was one of the
> > > > design
> > > > > > > > considerations of using flatbuffers and IPC Stream/File format.
> > > > > > > >
> > > > > > > > I was thinking Micah's comment is more that whatever we do, it
> > > > > should be
> > > > > > > > > clearly specified and edge cases should be considered,
> > > especially
> > > > > if we
> > > > > > > > > might want to 'backport' this into the stream format later.
> > > > > > > >
> > > > > > > >
> > > > > > > > Yes, for dictionaries we just need to be careful to define
> > > > semantics
> > > > > and
> > > > > > > > ensure implementations are validating them with regards to
> > > > > dictionaries.
> > > > > > > > There likely isn't any need to change current implementations
> > > > though.
> > > > > > > >
> > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Right, I had wanted to focus the discussion on Flight as I
> > > think
> > > > > schema
> > > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > > property of the
> > > > > > > > > transport and not the stream format itself. If we are leaning
> > > > > towards just
> > > > > > > > > schema evolution then maybe it makes sense to discuss it for
> > > the
> > > > > IPC stream
> > > > > > > > > format and leverage that in Flight. I'd be interested in what
> > > > > others think.
> > > > > > > > >
> > > > > > > > > Especially if we are looking at multiplexing streams - I would
> > > > > wonder if
> > > > > > > > > that's actually better served by making it easier to implement
> > > > > using the
> > > > > > > > > Flight implementation as it stands (by managing concurrent RPC
> > > > > calls and/or
> > > > > > > > > performing the union-of-structs encoding trick for you),
> > > instead
> > > > > of having
> > > > > > > > > to change the protocol.
> > > > > > > > >
> > > > > > > > > Nate: it may be worth starting a separate discussion about more
> > > > > general
> > > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > > metadata was
> > > > > > > > > chosen/if opaque bytes were considered in the past. Off the top
> > > > of
> > > > > my head
> > > > > > > > > if it's for on-disk storage and fully application-defined it
> > > may
> > > > > make sense
> > > > > > > > > to store as a separate file alongside the Arrow file (indexed
> > > by
> > > > > record
> > > > > > > > > batch index) where you can take advantage of whatever format is
> > > > > most
> > > > > > > > > suitable.
> > > > > > > > >
> > > > > > > > > -David
> > > > > > > > >
> > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > > > > Hi guys,
> > > > > > > > > >
> > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> > > > to
> > > > > add this
> > > > > > > > > > feature starting from the IPC(I moved initial write up steps
> > > to
> > > > > the
> > > > > > > > > bottom
> > > > > > > > > > of the doc). Afterwards David suggested focusing on Flight
> > > and
> > > > > that's how
> > > > > > > > > > we ended up with the protobufs change in the proposal. This
> > > > > being said I
> > > > > > > > > do
> > > > > > > > > > think that the place where this should be impemented is a
> > > good
> > > > > question
> > > > > > > > > on
> > > > > > > > > > its own. Maybe it makes sense to have this kind of a feature
> > > in
> > > > > IPC and
> > > > > > > > > > somehow use it in Flight, maybe not.
> > > > > > > > > > 2. The point about dictionaries deserves a dedicated section
> > > in
> > > > > the
> > > > > > > > > > proposal. Nate and David brought it up and shared some
> > > > insights.
> > > > > I'll try
> > > > > > > > > > to aggregate them and we can continue the discussion form
> > > > there.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Gosh
> > > > > > > > > >
> > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > > > > natebauernfeind@deephaven.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > > makes it more difficult to bring schema evolution
> > > back
> > > > > into the
> > > > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not
> > > the
> > > > > > > > > protobufs.
> > > > > > > > > > > Can
> > > > > > > > > > > > > you help me understand how difficult it would be to
> > > bring
> > > > > the
> > > > > > > > > > > `schema_id`
> > > > > > > > > > > > > approach to the IPC stream format?
> > > > > > > > > > > >
> > > > > > > > > > > > I thought we were talking solely about the Flight
> > > Protobuf
> > > > > > > > > definitions -
> > > > > > > > > > > > not the Flatbuffers (and the Google doc at least only
> > > talks
> > > > > about the
> > > > > > > > > > > > Protobufs).
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I somehow missed that schema_id is being added to protobuf
> > > in
> > > > > the
> > > > > > > > > document.
> > > > > > > > > > > It feels to me that the schema_id is a property that would
> > > > > ideally only
> > > > > > > > > > > apply to the RecordBatch. I better understand Micah's
> > > > > dictionary
> > > > > > > > > concerns,
> > > > > > > > > > > now, too.
> > > > > > > > > > >
> > > > > > > > > > > > Side Question: Why isn't the IPC stream format a series
> > > of
> > > > > the flight
> > > > > > > > > > > > > protobufs? It's a real shame that there is no standard
> > > > way
> > > > > to
> > > > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> > > > > ignoring the
> > > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > > > > >
> > > > > > > > > > > > The IPC format was defined long before Flight, and
> > > Flight's
> > > > > > > > > app_metadata
> > > > > > > > > > > > was added after Flight's initial definition. Note an IPC
> > > > > message does
> > > > > > > > > > > have
> > > > > > > > > > > > a provision for key-value metadata, though I think APIs
> > > for
> > > > > that are
> > > > > > > > > not
> > > > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > > > despite my
> > > > > > > > > comments
> > > > > > > > > > > > there perhaps we need to unify or at least consider how
> > > > > Flight's
> > > > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> > > > Also
> > > > > > > > > perhaps see
> > > > > > > > > > > > ARROW-1059.)
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> > > > > strings are
> > > > > > > > > only
> > > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> > > > > opaque
> > > > > > > > > bytes.
> > > > > > > > > > > The latter is a bit more useful.
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > >
> >
> >
> > --
> >

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

To summarize so far, it sounds like schema evolution is neither sufficient nor necessary for either Gosh or Nate's use-cases here? It could be useful for FlightSQL but even there I don't think it's a requirement.

For Nate - it almost sounds like what you need is some way to slice up a record batch and send columns individually, which isn't really a concept in IPC (and hence Flight). Or rather, record batch is almost the wrong abstraction for your use case (when you're sending per-column deltas), even though you could model it as a record batch with 'empty' columns or as a stream with constantly shifting schema (neither of which are perfect encodings). 

-David

On Wed, Jul 7, 2021, at 13:24, Nate Bauernfeind wrote:
> > Flatbuffers does not support modifying structs
> > in any forwards or backwards compatible way
> > (only tables support evolution).
> 
> Bah. I did not realize that.
> 
> To reiterate the feature that would be ideal:
> I realize the specific feature I am missing is the ability to encode that a
> field (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> empty/has-no-data in O(0) cost (yes; for free).
> 
> Since RecordBatch is a table, might there be interest in adding a field
> that is a bitset (formatted as a byte vector) to indicate the subset of
> root FieldNodes that are included in the RecordBatch's field-node-list
> (noting that the buffer list most be appropriately filtered, too)? If the
> bitset is omitted/empty we can be forwards-compatible by assuming every
> field is included (seems fair; because why send a record batch without a
> payload?). This does bring down the cost from 16 bytes (field node) + 32
> bytes (2x buffer node minimum; validity buffer node + payload buffer node)
> to 1 bit per field node -- that's a compression ratio of >= 384:1 --
> although it's not free, it's a lot closer.
> 
> Are there any other alternatives the Arrow community might consider?
> 
> 
> > This sounds a lot like what DenseUnion provides though?
> 
> The use case is as follows: We send multiple record batches that aggregate
> / accumulate into a single logical update. The first set of record batches
> contain payload for rows that were added since the last logical update
> (this is a set of updates to accommodate that not every implementation
> supports 32-bit lengths). A field node for every column and every added row
> will be sent. For this half of the logical update the RecordBatch length
> matches the root FieldNode lengths. The second set of record batches
> contain payload for only rows, and fields, that were actually modified. If
> only one field changed, we send only that payload for that row. There is
> additional metadata that allows the client to understand which existing row
> is to be replaced by any given row for a given Field/Column. In this
> context, each root field node has a different mapping to minimize total
> payload size.
> 
> We might be able to use DenseUnion but it certainly feels like folding the
> entire data-source into a dense union makes the design less useful and less
> accessible. I will spend some time sleeping on your suggestion, but I'm not
> immediately excited about it. At this moment, I suspect I will continue to
> lie and state that the RecordBatch's length is the max length across all
> root field node lengths (and be content that it's not ideal from a
> copy/allocation perspective).
> -
> 
> On Wed, Jul 7, 2021 at 10:57 AM Micah Kornfield <em...@gmail.com>
> wrote:
> 
> > >
> > > Might there be interest in adding a "field_id" to the FieldNode (which is
> > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > > upgrade (by either keying off of 0, or explicitly set the field default
> > to
> > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > > length of zero, and 2) all Buffer's associated at that level (and further
> > > nested) are also equally empty (i.e. Buffer length is zero).
> >
> >
> > FieldNode is a struct in Message.fbs.   Flatbuffers does not support
> > modifying structs in any forwards or backwards compatible way (only tables
> > support evolution).  I think there was originally more metadata in
> > FieldNode and it was stripped down due to size concerns.
> >
> > I understand this concept slightly interferes with RecordBatch's `length`
> > > field, and that many implementations use that length to resize the
> > > root-level FieldNodes. The use-case I have in mind has different logical
> > > lengths per field node; current implementations require sending a
> > > RecordBatch length of the max length across all root level field nodes. I
> > > believe this requires a copy of data whenever a field node is too short;
> > I
> > > don't know if there is a decent solution to this slight inefficiency. I
> > am
> > > bringing it up because if "skipping a field node when it is empty" is a
> > > feature, then we may not want to allocate space for those nodes given
> > that
> > > the record batch length will likely be greater than zero.
> >
> >
> > Having conflicting RecordBatch and top-level field nodes is something that
> > I believe we have pushed back on in the past.   This sounds a lot like
> > what  DenseUnion provides though?
> >
> > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> > natebauernfeind@deephaven.io> wrote:
> >
> > > Deephaven and I are very supportive of "upgrading" the value half of the
> > kv
> > > pair to a byte vector. What is the best way to find out if there is
> > > sufficient interest?
> > >
> > >
> > > I've been stewing on the ideas here around schema evolution, and I
> > realize
> > > the specific feature I am missing is the ability to encode that a field
> > > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > > empty/has-no-data in O(0) cost (yes; for free).
> > >
> > > Might there be interest in adding a "field_id" to the FieldNode (which is
> > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > > upgrade (by either keying off of 0, or explicitly set the field default
> > to
> > > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > > length of zero, and 2) all Buffer's associated at that level (and further
> > > nested) are also equally empty (i.e. Buffer length is zero).
> > >
> > > I understand this concept slightly interferes with RecordBatch's `length`
> > > field, and that many implementations use that length to resize the
> > > root-level FieldNodes. The use-case I have in mind has different logical
> > > lengths per field node; current implementations require sending a
> > > RecordBatch length of the max length across all root level field nodes. I
> > > believe this requires a copy of data whenever a field node is too short;
> > I
> > > don't know if there is a decent solution to this slight inefficiency. I
> > am
> > > bringing it up because if "skipping a field node when it is empty" is a
> > > feature, then we may not want to allocate space for those nodes given
> > that
> > > the record batch length will likely be greater than zero.
> > >
> > > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
> > >
> > > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> > > > >
> > > > > From the Flatbuffers internals doc[1] it appears they are the same:
> > > > "Strings are simply a vector of bytes, and are always null-terminated."
> > > >
> > > > I see. I took a look at flatbuffers.h, and it appears that changing
> > > > this field from string to [byte] would be backward-compatible and
> > > > forward-compatible except with code that expects a null terminator.
> > > > This is something we could discuss separately if there were enough
> > > > interest.
> > > >
> > > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > > >
> > > > > -David
> > > > >
> > > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > > emkornfield@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Right, I had wanted to focus the discussion on Flight as I
> > think
> > > > schema
> > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > property of the
> > > > > > > > transport and not the stream format itself. If we are leaning
> > > > towards just
> > > > > > > > schema evolution then maybe it makes sense to discuss it for
> > the
> > > > IPC stream
> > > > > > > > format and leverage that in Flight. I'd be interested in what
> > > > others think.
> > > > > > >
> > > > > > > I tend to agree, I think stream multiplexing is likely a
> > transport
> > > > level
> > > > > > > issue.  IMO I think schema evolution should be consistent with
> > the
> > > > IPC
> > > > > > > stream format  and flight.
> > > > > > >
> > > > > > >
> > > > > > > > Nate: it may be worth starting a separate discussion about more
> > > > general
> > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > metadata was
> > > > > > > > chosen/if opaque bytes were considered in the past.
> > > > > > >
> > > > > > >
> > > > > > > I think  this was an unfortunate design of the key value metadata
> > > in
> > > > > > > Schema.fbs, but I don't think I was around when this decision was
> > > > made.
> > > > > >
> > > > > > I agree that it's unfortunate that we did not use [ byte ] instead
> > of
> > > > > > string for the value in the KeyValue metadata — I think this was
> > more
> > > > > > of an oversight than a deliberate choice (e.g. it was not our
> > intent
> > > > > > to require binary data to be base64-encoded — this is something
> > that
> > > > > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > > > > for Parquet, for example). Is the binary representation of [byte]
> > > > > > different from string?
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Side Question: Why isn't the IPC stream format a series of the
> > > flight
> > > > > > > > protobufs?
> > > > > > >
> > > > > > > In addition to what David said, protobufs can't be read directly
> > > > from a
> > > > > > > memory-mapped file (they need decoding).  This was one of the
> > > design
> > > > > > > considerations of using flatbuffers and IPC Stream/File format.
> > > > > > >
> > > > > > > I was thinking Micah's comment is more that whatever we do, it
> > > > should be
> > > > > > > > clearly specified and edge cases should be considered,
> > especially
> > > > if we
> > > > > > > > might want to 'backport' this into the stream format later.
> > > > > > >
> > > > > > >
> > > > > > > Yes, for dictionaries we just need to be careful to define
> > > semantics
> > > > and
> > > > > > > ensure implementations are validating them with regards to
> > > > dictionaries.
> > > > > > > There likely isn't any need to change current implementations
> > > though.
> > > > > > >
> > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Right, I had wanted to focus the discussion on Flight as I
> > think
> > > > schema
> > > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > > property of the
> > > > > > > > transport and not the stream format itself. If we are leaning
> > > > towards just
> > > > > > > > schema evolution then maybe it makes sense to discuss it for
> > the
> > > > IPC stream
> > > > > > > > format and leverage that in Flight. I'd be interested in what
> > > > others think.
> > > > > > > >
> > > > > > > > Especially if we are looking at multiplexing streams - I would
> > > > wonder if
> > > > > > > > that's actually better served by making it easier to implement
> > > > using the
> > > > > > > > Flight implementation as it stands (by managing concurrent RPC
> > > > calls and/or
> > > > > > > > performing the union-of-structs encoding trick for you),
> > instead
> > > > of having
> > > > > > > > to change the protocol.
> > > > > > > >
> > > > > > > > Nate: it may be worth starting a separate discussion about more
> > > > general
> > > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > > metadata was
> > > > > > > > chosen/if opaque bytes were considered in the past. Off the top
> > > of
> > > > my head
> > > > > > > > if it's for on-disk storage and fully application-defined it
> > may
> > > > make sense
> > > > > > > > to store as a separate file alongside the Arrow file (indexed
> > by
> > > > record
> > > > > > > > batch index) where you can take advantage of whatever format is
> > > > most
> > > > > > > > suitable.
> > > > > > > >
> > > > > > > > -David
> > > > > > > >
> > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> > > to
> > > > add this
> > > > > > > > > feature starting from the IPC(I moved initial write up steps
> > to
> > > > the
> > > > > > > > bottom
> > > > > > > > > of the doc). Afterwards David suggested focusing on Flight
> > and
> > > > that's how
> > > > > > > > > we ended up with the protobufs change in the proposal. This
> > > > being said I
> > > > > > > > do
> > > > > > > > > think that the place where this should be impemented is a
> > good
> > > > question
> > > > > > > > on
> > > > > > > > > its own. Maybe it makes sense to have this kind of a feature
> > in
> > > > IPC and
> > > > > > > > > somehow use it in Flight, maybe not.
> > > > > > > > > 2. The point about dictionaries deserves a dedicated section
> > in
> > > > the
> > > > > > > > > proposal. Nate and David brought it up and shared some
> > > insights.
> > > > I'll try
> > > > > > > > > to aggregate them and we can continue the discussion form
> > > there.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Gosh
> > > > > > > > >
> > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > > > natebauernfeind@deephaven.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > makes it more difficult to bring schema evolution
> > back
> > > > into the
> > > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > > > > >
> > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not
> > the
> > > > > > > > protobufs.
> > > > > > > > > > Can
> > > > > > > > > > > > you help me understand how difficult it would be to
> > bring
> > > > the
> > > > > > > > > > `schema_id`
> > > > > > > > > > > > approach to the IPC stream format?
> > > > > > > > > > >
> > > > > > > > > > > I thought we were talking solely about the Flight
> > Protobuf
> > > > > > > > definitions -
> > > > > > > > > > > not the Flatbuffers (and the Google doc at least only
> > talks
> > > > about the
> > > > > > > > > > > Protobufs).
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I somehow missed that schema_id is being added to protobuf
> > in
> > > > the
> > > > > > > > document.
> > > > > > > > > > It feels to me that the schema_id is a property that would
> > > > ideally only
> > > > > > > > > > apply to the RecordBatch. I better understand Micah's
> > > > dictionary
> > > > > > > > concerns,
> > > > > > > > > > now, too.
> > > > > > > > > >
> > > > > > > > > > > Side Question: Why isn't the IPC stream format a series
> > of
> > > > the flight
> > > > > > > > > > > > protobufs? It's a real shame that there is no standard
> > > way
> > > > to
> > > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> > > > ignoring the
> > > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > > > >
> > > > > > > > > > > The IPC format was defined long before Flight, and
> > Flight's
> > > > > > > > app_metadata
> > > > > > > > > > > was added after Flight's initial definition. Note an IPC
> > > > message does
> > > > > > > > > > have
> > > > > > > > > > > a provision for key-value metadata, though I think APIs
> > for
> > > > that are
> > > > > > > > not
> > > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > > despite my
> > > > > > > > comments
> > > > > > > > > > > there perhaps we need to unify or at least consider how
> > > > Flight's
> > > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> > > Also
> > > > > > > > perhaps see
> > > > > > > > > > > ARROW-1059.)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> > > > strings are
> > > > > > > > only
> > > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> > > > opaque
> > > > > > > > bytes.
> > > > > > > > > > The latter is a bit more useful.
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
> > >
> > > --
> > >
> >
> 
> 
> --
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Nate Bauernfeind <na...@deephaven.io>]

> Flatbuffers does not support modifying structs
> in any forwards or backwards compatible way
> (only tables support evolution).

Bah. I did not realize that.

To reiterate the feature that would be ideal:
I realize the specific feature I am missing is the ability to encode that a
field (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
empty/has-no-data in O(0) cost (yes; for free).

Since RecordBatch is a table, might there be interest in adding a field
that is a bitset (formatted as a byte vector) to indicate the subset of
root FieldNodes that are included in the RecordBatch's field-node-list
(noting that the buffer list most be appropriately filtered, too)? If the
bitset is omitted/empty we can be forwards-compatible by assuming every
field is included (seems fair; because why send a record batch without a
payload?). This does bring down the cost from 16 bytes (field node) + 32
bytes (2x buffer node minimum; validity buffer node + payload buffer node)
to 1 bit per field node -- that's a compression ratio of >= 384:1 --
although it's not free, it's a lot closer.

Are there any other alternatives the Arrow community might consider?


> This sounds a lot like what DenseUnion provides though?

The use case is as follows: We send multiple record batches that aggregate
/ accumulate into a single logical update. The first set of record batches
contain payload for rows that were added since the last logical update
(this is a set of updates to accommodate that not every implementation
supports 32-bit lengths). A field node for every column and every added row
will be sent. For this half of the logical update the RecordBatch length
matches the root FieldNode lengths. The second set of record batches
contain payload for only rows, and fields, that were actually modified. If
only one field changed, we send only that payload for that row. There is
additional metadata that allows the client to understand which existing row
is to be replaced by any given row for a given Field/Column. In this
context, each root field node has a different mapping to minimize total
payload size.

We might be able to use DenseUnion but it certainly feels like folding the
entire data-source into a dense union makes the design less useful and less
accessible. I will spend some time sleeping on your suggestion, but I'm not
immediately excited about it. At this moment, I suspect I will continue to
lie and state that the RecordBatch's length is the max length across all
root field node lengths (and be content that it's not ideal from a
copy/allocation perspective).
-

On Wed, Jul 7, 2021 at 10:57 AM Micah Kornfield <em...@gmail.com>
wrote:

> >
> > Might there be interest in adding a "field_id" to the FieldNode (which is
> > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > upgrade (by either keying off of 0, or explicitly set the field default
> to
> > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > length of zero, and 2) all Buffer's associated at that level (and further
> > nested) are also equally empty (i.e. Buffer length is zero).
>
>
> FieldNode is a struct in Message.fbs.   Flatbuffers does not support
> modifying structs in any forwards or backwards compatible way (only tables
> support evolution).  I think there was originally more metadata in
> FieldNode and it was stripped down due to size concerns.
>
> I understand this concept slightly interferes with RecordBatch's `length`
> > field, and that many implementations use that length to resize the
> > root-level FieldNodes. The use-case I have in mind has different logical
> > lengths per field node; current implementations require sending a
> > RecordBatch length of the max length across all root level field nodes. I
> > believe this requires a copy of data whenever a field node is too short;
> I
> > don't know if there is a decent solution to this slight inefficiency. I
> am
> > bringing it up because if "skipping a field node when it is empty" is a
> > feature, then we may not want to allocate space for those nodes given
> that
> > the record batch length will likely be greater than zero.
>
>
> Having conflicting RecordBatch and top-level field nodes is something that
> I believe we have pushed back on in the past.   This sounds a lot like
> what  DenseUnion provides though?
>
> On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
> natebauernfeind@deephaven.io> wrote:
>
> > Deephaven and I are very supportive of "upgrading" the value half of the
> kv
> > pair to a byte vector. What is the best way to find out if there is
> > sufficient interest?
> >
> >
> > I've been stewing on the ideas here around schema evolution, and I
> realize
> > the specific feature I am missing is the ability to encode that a field
> > (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> > empty/has-no-data in O(0) cost (yes; for free).
> >
> > Might there be interest in adding a "field_id" to the FieldNode (which is
> > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> > upgrade (by either keying off of 0, or explicitly set the field default
> to
> > -1) which would allow the sender to "skip" fields that have 1) FieldNode
> > length of zero, and 2) all Buffer's associated at that level (and further
> > nested) are also equally empty (i.e. Buffer length is zero).
> >
> > I understand this concept slightly interferes with RecordBatch's `length`
> > field, and that many implementations use that length to resize the
> > root-level FieldNodes. The use-case I have in mind has different logical
> > lengths per field node; current implementations require sending a
> > RecordBatch length of the max length across all root level field nodes. I
> > believe this requires a copy of data whenever a field node is too short;
> I
> > don't know if there is a decent solution to this slight inefficiency. I
> am
> > bringing it up because if "skipping a field node when it is empty" is a
> > feature, then we may not want to allocate space for those nodes given
> that
> > the record batch length will likely be greater than zero.
> >
> > On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
> >
> > > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> > > >
> > > > From the Flatbuffers internals doc[1] it appears they are the same:
> > > "Strings are simply a vector of bytes, and are always null-terminated."
> > >
> > > I see. I took a look at flatbuffers.h, and it appears that changing
> > > this field from string to [byte] would be backward-compatible and
> > > forward-compatible except with code that expects a null terminator.
> > > This is something we could discuss separately if there were enough
> > > interest.
> > >
> > > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> > > >
> > > > -David
> > > >
> > > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Right, I had wanted to focus the discussion on Flight as I
> think
> > > schema
> > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > property of the
> > > > > > > transport and not the stream format itself. If we are leaning
> > > towards just
> > > > > > > schema evolution then maybe it makes sense to discuss it for
> the
> > > IPC stream
> > > > > > > format and leverage that in Flight. I'd be interested in what
> > > others think.
> > > > > >
> > > > > > I tend to agree, I think stream multiplexing is likely a
> transport
> > > level
> > > > > > issue.  IMO I think schema evolution should be consistent with
> the
> > > IPC
> > > > > > stream format  and flight.
> > > > > >
> > > > > >
> > > > > > > Nate: it may be worth starting a separate discussion about more
> > > general
> > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > metadata was
> > > > > > > chosen/if opaque bytes were considered in the past.
> > > > > >
> > > > > >
> > > > > > I think  this was an unfortunate design of the key value metadata
> > in
> > > > > > Schema.fbs, but I don't think I was around when this decision was
> > > made.
> > > > >
> > > > > I agree that it's unfortunate that we did not use [ byte ] instead
> of
> > > > > string for the value in the KeyValue metadata — I think this was
> more
> > > > > of an oversight than a deliberate choice (e.g. it was not our
> intent
> > > > > to require binary data to be base64-encoded — this is something
> that
> > > > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > > > for Parquet, for example). Is the binary representation of [byte]
> > > > > different from string?
> > > > >
> > > > >
> > > > >
> > > > > > Side Question: Why isn't the IPC stream format a series of the
> > flight
> > > > > > > protobufs?
> > > > > >
> > > > > > In addition to what David said, protobufs can't be read directly
> > > from a
> > > > > > memory-mapped file (they need decoding).  This was one of the
> > design
> > > > > > considerations of using flatbuffers and IPC Stream/File format.
> > > > > >
> > > > > > I was thinking Micah's comment is more that whatever we do, it
> > > should be
> > > > > > > clearly specified and edge cases should be considered,
> especially
> > > if we
> > > > > > > might want to 'backport' this into the stream format later.
> > > > > >
> > > > > >
> > > > > > Yes, for dictionaries we just need to be careful to define
> > semantics
> > > and
> > > > > > ensure implementations are validating them with regards to
> > > dictionaries.
> > > > > > There likely isn't any need to change current implementations
> > though.
> > > > > >
> > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Right, I had wanted to focus the discussion on Flight as I
> think
> > > schema
> > > > > > > evolution or multiplexing streams (more so the latter) is a
> > > property of the
> > > > > > > transport and not the stream format itself. If we are leaning
> > > towards just
> > > > > > > schema evolution then maybe it makes sense to discuss it for
> the
> > > IPC stream
> > > > > > > format and leverage that in Flight. I'd be interested in what
> > > others think.
> > > > > > >
> > > > > > > Especially if we are looking at multiplexing streams - I would
> > > wonder if
> > > > > > > that's actually better served by making it easier to implement
> > > using the
> > > > > > > Flight implementation as it stands (by managing concurrent RPC
> > > calls and/or
> > > > > > > performing the union-of-structs encoding trick for you),
> instead
> > > of having
> > > > > > > to change the protocol.
> > > > > > >
> > > > > > > Nate: it may be worth starting a separate discussion about more
> > > general
> > > > > > > metadata in the IPC message. I'm not aware of why key-value
> > > metadata was
> > > > > > > chosen/if opaque bytes were considered in the past. Off the top
> > of
> > > my head
> > > > > > > if it's for on-disk storage and fully application-defined it
> may
> > > make sense
> > > > > > > to store as a separate file alongside the Arrow file (indexed
> by
> > > record
> > > > > > > batch index) where you can take advantage of whatever format is
> > > most
> > > > > > > suitable.
> > > > > > >
> > > > > > > -David
> > > > > > >
> > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> > to
> > > add this
> > > > > > > > feature starting from the IPC(I moved initial write up steps
> to
> > > the
> > > > > > > bottom
> > > > > > > > of the doc). Afterwards David suggested focusing on Flight
> and
> > > that's how
> > > > > > > > we ended up with the protobufs change in the proposal. This
> > > being said I
> > > > > > > do
> > > > > > > > think that the place where this should be impemented is a
> good
> > > question
> > > > > > > on
> > > > > > > > its own. Maybe it makes sense to have this kind of a feature
> in
> > > IPC and
> > > > > > > > somehow use it in Flight, maybe not.
> > > > > > > > 2. The point about dictionaries deserves a dedicated section
> in
> > > the
> > > > > > > > proposal. Nate and David brought it up and shared some
> > insights.
> > > I'll try
> > > > > > > > to aggregate them and we can continue the discussion form
> > there.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Gosh
> > > > > > > >
> > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > > natebauernfeind@deephaven.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > makes it more difficult to bring schema evolution
> back
> > > into the
> > > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > > > >
> > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not
> the
> > > > > > > protobufs.
> > > > > > > > > Can
> > > > > > > > > > > you help me understand how difficult it would be to
> bring
> > > the
> > > > > > > > > `schema_id`
> > > > > > > > > > > approach to the IPC stream format?
> > > > > > > > > >
> > > > > > > > > > I thought we were talking solely about the Flight
> Protobuf
> > > > > > > definitions -
> > > > > > > > > > not the Flatbuffers (and the Google doc at least only
> talks
> > > about the
> > > > > > > > > > Protobufs).
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I somehow missed that schema_id is being added to protobuf
> in
> > > the
> > > > > > > document.
> > > > > > > > > It feels to me that the schema_id is a property that would
> > > ideally only
> > > > > > > > > apply to the RecordBatch. I better understand Micah's
> > > dictionary
> > > > > > > concerns,
> > > > > > > > > now, too.
> > > > > > > > >
> > > > > > > > > > Side Question: Why isn't the IPC stream format a series
> of
> > > the flight
> > > > > > > > > > > protobufs? It's a real shame that there is no standard
> > way
> > > to
> > > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> > > ignoring the
> > > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > > >
> > > > > > > > > > The IPC format was defined long before Flight, and
> Flight's
> > > > > > > app_metadata
> > > > > > > > > > was added after Flight's initial definition. Note an IPC
> > > message does
> > > > > > > > > have
> > > > > > > > > > a provision for key-value metadata, though I think APIs
> for
> > > that are
> > > > > > > not
> > > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > > despite my
> > > > > > > comments
> > > > > > > > > > there perhaps we need to unify or at least consider how
> > > Flight's
> > > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> > Also
> > > > > > > perhaps see
> > > > > > > > > > ARROW-1059.)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> > > strings are
> > > > > > > only
> > > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> > > opaque
> > > > > > > bytes.
> > > > > > > > > The latter is a bit more useful.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> >
> > --
> >
>


--

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

>
> Might there be interest in adding a "field_id" to the FieldNode (which is
> encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> upgrade (by either keying off of 0, or explicitly set the field default to
> -1) which would allow the sender to "skip" fields that have 1) FieldNode
> length of zero, and 2) all Buffer's associated at that level (and further
> nested) are also equally empty (i.e. Buffer length is zero).


FieldNode is a struct in Message.fbs.   Flatbuffers does not support
modifying structs in any forwards or backwards compatible way (only tables
support evolution).  I think there was originally more metadata in
FieldNode and it was stripped down due to size concerns.

I understand this concept slightly interferes with RecordBatch's `length`
> field, and that many implementations use that length to resize the
> root-level FieldNodes. The use-case I have in mind has different logical
> lengths per field node; current implementations require sending a
> RecordBatch length of the max length across all root level field nodes. I
> believe this requires a copy of data whenever a field node is too short; I
> don't know if there is a decent solution to this slight inefficiency. I am
> bringing it up because if "skipping a field node when it is empty" is a
> feature, then we may not want to allocate space for those nodes given that
> the record batch length will likely be greater than zero.


Having conflicting RecordBatch and top-level field nodes is something that
I believe we have pushed back on in the past.   This sounds a lot like
what  DenseUnion provides though?

On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind <
natebauernfeind@deephaven.io> wrote:

> Deephaven and I are very supportive of "upgrading" the value half of the kv
> pair to a byte vector. What is the best way to find out if there is
> sufficient interest?
>
>
> I've been stewing on the ideas here around schema evolution, and I realize
> the specific feature I am missing is the ability to encode that a field
> (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
> empty/has-no-data in O(0) cost (yes; for free).
>
> Might there be interest in adding a "field_id" to the FieldNode (which is
> encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
> upgrade (by either keying off of 0, or explicitly set the field default to
> -1) which would allow the sender to "skip" fields that have 1) FieldNode
> length of zero, and 2) all Buffer's associated at that level (and further
> nested) are also equally empty (i.e. Buffer length is zero).
>
> I understand this concept slightly interferes with RecordBatch's `length`
> field, and that many implementations use that length to resize the
> root-level FieldNodes. The use-case I have in mind has different logical
> lengths per field node; current implementations require sending a
> RecordBatch length of the max length across all root level field nodes. I
> believe this requires a copy of data whenever a field node is too short; I
> don't know if there is a decent solution to this slight inefficiency. I am
> bringing it up because if "skipping a field node when it is empty" is a
> feature, then we may not want to allocate space for those nodes given that
> the record batch length will likely be greater than zero.
>
> On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:
>
> > On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> > >
> > > From the Flatbuffers internals doc[1] it appears they are the same:
> > "Strings are simply a vector of bytes, and are always null-terminated."
> >
> > I see. I took a look at flatbuffers.h, and it appears that changing
> > this field from string to [byte] would be backward-compatible and
> > forward-compatible except with code that expects a null terminator.
> > This is something we could discuss separately if there were enough
> > interest.
> >
> > > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> > >
> > > -David
> > >
> > > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <
> emkornfield@gmail.com>
> > wrote:
> > > > >
> > > > > >
> > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > schema
> > > > > > evolution or multiplexing streams (more so the latter) is a
> > property of the
> > > > > > transport and not the stream format itself. If we are leaning
> > towards just
> > > > > > schema evolution then maybe it makes sense to discuss it for the
> > IPC stream
> > > > > > format and leverage that in Flight. I'd be interested in what
> > others think.
> > > > >
> > > > > I tend to agree, I think stream multiplexing is likely a transport
> > level
> > > > > issue.  IMO I think schema evolution should be consistent with the
> > IPC
> > > > > stream format  and flight.
> > > > >
> > > > >
> > > > > > Nate: it may be worth starting a separate discussion about more
> > general
> > > > > > metadata in the IPC message. I'm not aware of why key-value
> > metadata was
> > > > > > chosen/if opaque bytes were considered in the past.
> > > > >
> > > > >
> > > > > I think  this was an unfortunate design of the key value metadata
> in
> > > > > Schema.fbs, but I don't think I was around when this decision was
> > made.
> > > >
> > > > I agree that it's unfortunate that we did not use [ byte ] instead of
> > > > string for the value in the KeyValue metadata — I think this was more
> > > > of an oversight than a deliberate choice (e.g. it was not our intent
> > > > to require binary data to be base64-encoded — this is something that
> > > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > > for Parquet, for example). Is the binary representation of [byte]
> > > > different from string?
> > > >
> > > >
> > > >
> > > > > Side Question: Why isn't the IPC stream format a series of the
> flight
> > > > > > protobufs?
> > > > >
> > > > > In addition to what David said, protobufs can't be read directly
> > from a
> > > > > memory-mapped file (they need decoding).  This was one of the
> design
> > > > > considerations of using flatbuffers and IPC Stream/File format.
> > > > >
> > > > > I was thinking Micah's comment is more that whatever we do, it
> > should be
> > > > > > clearly specified and edge cases should be considered, especially
> > if we
> > > > > > might want to 'backport' this into the stream format later.
> > > > >
> > > > >
> > > > > Yes, for dictionaries we just need to be careful to define
> semantics
> > and
> > > > > ensure implementations are validating them with regards to
> > dictionaries.
> > > > > There likely isn't any need to change current implementations
> though.
> > > > >
> > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> > wrote:
> > > > >
> > > > > > Right, I had wanted to focus the discussion on Flight as I think
> > schema
> > > > > > evolution or multiplexing streams (more so the latter) is a
> > property of the
> > > > > > transport and not the stream format itself. If we are leaning
> > towards just
> > > > > > schema evolution then maybe it makes sense to discuss it for the
> > IPC stream
> > > > > > format and leverage that in Flight. I'd be interested in what
> > others think.
> > > > > >
> > > > > > Especially if we are looking at multiplexing streams - I would
> > wonder if
> > > > > > that's actually better served by making it easier to implement
> > using the
> > > > > > Flight implementation as it stands (by managing concurrent RPC
> > calls and/or
> > > > > > performing the union-of-structs encoding trick for you), instead
> > of having
> > > > > > to change the protocol.
> > > > > >
> > > > > > Nate: it may be worth starting a separate discussion about more
> > general
> > > > > > metadata in the IPC message. I'm not aware of why key-value
> > metadata was
> > > > > > chosen/if opaque bytes were considered in the past. Off the top
> of
> > my head
> > > > > > if it's for on-disk storage and fully application-defined it may
> > make sense
> > > > > > to store as a separate file alongside the Arrow file (indexed by
> > record
> > > > > > batch index) where you can take advantage of whatever format is
> > most
> > > > > > suitable.
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was
> to
> > add this
> > > > > > > feature starting from the IPC(I moved initial write up steps to
> > the
> > > > > > bottom
> > > > > > > of the doc). Afterwards David suggested focusing on Flight and
> > that's how
> > > > > > > we ended up with the protobufs change in the proposal. This
> > being said I
> > > > > > do
> > > > > > > think that the place where this should be impemented is a good
> > question
> > > > > > on
> > > > > > > its own. Maybe it makes sense to have this kind of a feature in
> > IPC and
> > > > > > > somehow use it in Flight, maybe not.
> > > > > > > 2. The point about dictionaries deserves a dedicated section in
> > the
> > > > > > > proposal. Nate and David brought it up and shared some
> insights.
> > I'll try
> > > > > > > to aggregate them and we can continue the discussion form
> there.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Gosh
> > > > > > >
> > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > > natebauernfeind@deephaven.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > >
> > > > > > > > > > > makes it more difficult to bring schema evolution back
> > into the
> > > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > > >
> > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
> > > > > > protobufs.
> > > > > > > > Can
> > > > > > > > > > you help me understand how difficult it would be to bring
> > the
> > > > > > > > `schema_id`
> > > > > > > > > > approach to the IPC stream format?
> > > > > > > > >
> > > > > > > > > I thought we were talking solely about the Flight Protobuf
> > > > > > definitions -
> > > > > > > > > not the Flatbuffers (and the Google doc at least only talks
> > about the
> > > > > > > > > Protobufs).
> > > > > > > > >
> > > > > > > >
> > > > > > > > I somehow missed that schema_id is being added to protobuf in
> > the
> > > > > > document.
> > > > > > > > It feels to me that the schema_id is a property that would
> > ideally only
> > > > > > > > apply to the RecordBatch. I better understand Micah's
> > dictionary
> > > > > > concerns,
> > > > > > > > now, too.
> > > > > > > >
> > > > > > > > > Side Question: Why isn't the IPC stream format a series of
> > the flight
> > > > > > > > > > protobufs? It's a real shame that there is no standard
> way
> > to
> > > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> > ignoring the
> > > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > > >
> > > > > > > > > The IPC format was defined long before Flight, and Flight's
> > > > > > app_metadata
> > > > > > > > > was added after Flight's initial definition. Note an IPC
> > message does
> > > > > > > > have
> > > > > > > > > a provision for key-value metadata, though I think APIs for
> > that are
> > > > > > not
> > > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> > despite my
> > > > > > comments
> > > > > > > > > there perhaps we need to unify or at least consider how
> > Flight's
> > > > > > > > > app_metadata relates to the IPC message custom_metadata.
> Also
> > > > > > perhaps see
> > > > > > > > > ARROW-1059.)
> > > > > > > > >
> > > > > > > >
> > > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> > strings are
> > > > > > only
> > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> > opaque
> > > > > > bytes.
> > > > > > > > The latter is a bit more useful.
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>
>
> --
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Nate Bauernfeind <na...@deephaven.io>]

Deephaven and I are very supportive of "upgrading" the value half of the kv
pair to a byte vector. What is the best way to find out if there is
sufficient interest?


I've been stewing on the ideas here around schema evolution, and I realize
the specific feature I am missing is the ability to encode that a field
(i.e. its FieldNode and accompanying Buffers in the RecordBatch) is
empty/has-no-data in O(0) cost (yes; for free).

Might there be interest in adding a "field_id" to the FieldNode (which is
encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible
upgrade (by either keying off of 0, or explicitly set the field default to
-1) which would allow the sender to "skip" fields that have 1) FieldNode
length of zero, and 2) all Buffer's associated at that level (and further
nested) are also equally empty (i.e. Buffer length is zero).

I understand this concept slightly interferes with RecordBatch's `length`
field, and that many implementations use that length to resize the
root-level FieldNodes. The use-case I have in mind has different logical
lengths per field node; current implementations require sending a
RecordBatch length of the max length across all root level field nodes. I
believe this requires a copy of data whenever a field node is too short; I
don't know if there is a decent solution to this slight inefficiency. I am
bringing it up because if "skipping a field node when it is empty" is a
feature, then we may not want to allocate space for those nodes given that
the record batch length will likely be greater than zero.

On Wed, Jul 7, 2021 at 8:12 AM Wes McKinney <we...@gmail.com> wrote:

> On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
> >
> > From the Flatbuffers internals doc[1] it appears they are the same:
> "Strings are simply a vector of bytes, and are always null-terminated."
>
> I see. I took a look at flatbuffers.h, and it appears that changing
> this field from string to [byte] would be backward-compatible and
> forward-compatible except with code that expects a null terminator.
> This is something we could discuss separately if there were enough
> interest.
>
> > [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
> >
> > -David
> >
> > On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <em...@gmail.com>
> wrote:
> > > >
> > > > >
> > > > > Right, I had wanted to focus the discussion on Flight as I think
> schema
> > > > > evolution or multiplexing streams (more so the latter) is a
> property of the
> > > > > transport and not the stream format itself. If we are leaning
> towards just
> > > > > schema evolution then maybe it makes sense to discuss it for the
> IPC stream
> > > > > format and leverage that in Flight. I'd be interested in what
> others think.
> > > >
> > > > I tend to agree, I think stream multiplexing is likely a transport
> level
> > > > issue.  IMO I think schema evolution should be consistent with the
> IPC
> > > > stream format  and flight.
> > > >
> > > >
> > > > > Nate: it may be worth starting a separate discussion about more
> general
> > > > > metadata in the IPC message. I'm not aware of why key-value
> metadata was
> > > > > chosen/if opaque bytes were considered in the past.
> > > >
> > > >
> > > > I think  this was an unfortunate design of the key value metadata in
> > > > Schema.fbs, but I don't think I was around when this decision was
> made.
> > >
> > > I agree that it's unfortunate that we did not use [ byte ] instead of
> > > string for the value in the KeyValue metadata — I think this was more
> > > of an oversight than a deliberate choice (e.g. it was not our intent
> > > to require binary data to be base64-encoded — this is something that
> > > we have to do when encoding binary data in Thrift KeyValue metadata
> > > for Parquet, for example). Is the binary representation of [byte]
> > > different from string?
> > >
> > >
> > >
> > > > Side Question: Why isn't the IPC stream format a series of the flight
> > > > > protobufs?
> > > >
> > > > In addition to what David said, protobufs can't be read directly
> from a
> > > > memory-mapped file (they need decoding).  This was one of the design
> > > > considerations of using flatbuffers and IPC Stream/File format.
> > > >
> > > > I was thinking Micah's comment is more that whatever we do, it
> should be
> > > > > clearly specified and edge cases should be considered, especially
> if we
> > > > > might want to 'backport' this into the stream format later.
> > > >
> > > >
> > > > Yes, for dictionaries we just need to be careful to define semantics
> and
> > > > ensure implementations are validating them with regards to
> dictionaries.
> > > > There likely isn't any need to change current implementations though.
> > > >
> > > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org>
> wrote:
> > > >
> > > > > Right, I had wanted to focus the discussion on Flight as I think
> schema
> > > > > evolution or multiplexing streams (more so the latter) is a
> property of the
> > > > > transport and not the stream format itself. If we are leaning
> towards just
> > > > > schema evolution then maybe it makes sense to discuss it for the
> IPC stream
> > > > > format and leverage that in Flight. I'd be interested in what
> others think.
> > > > >
> > > > > Especially if we are looking at multiplexing streams - I would
> wonder if
> > > > > that's actually better served by making it easier to implement
> using the
> > > > > Flight implementation as it stands (by managing concurrent RPC
> calls and/or
> > > > > performing the union-of-structs encoding trick for you), instead
> of having
> > > > > to change the protocol.
> > > > >
> > > > > Nate: it may be worth starting a separate discussion about more
> general
> > > > > metadata in the IPC message. I'm not aware of why key-value
> metadata was
> > > > > chosen/if opaque bytes were considered in the past. Off the top of
> my head
> > > > > if it's for on-disk storage and fully application-defined it may
> make sense
> > > > > to store as a separate file alongside the Arrow file (indexed by
> record
> > > > > batch index) where you can take advantage of whatever format is
> most
> > > > > suitable.
> > > > >
> > > > > -David
> > > > >
> > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > > Hi guys,
> > > > > >
> > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was to
> add this
> > > > > > feature starting from the IPC(I moved initial write up steps to
> the
> > > > > bottom
> > > > > > of the doc). Afterwards David suggested focusing on Flight and
> that's how
> > > > > > we ended up with the protobufs change in the proposal. This
> being said I
> > > > > do
> > > > > > think that the place where this should be impemented is a good
> question
> > > > > on
> > > > > > its own. Maybe it makes sense to have this kind of a feature in
> IPC and
> > > > > > somehow use it in Flight, maybe not.
> > > > > > 2. The point about dictionaries deserves a dedicated section in
> the
> > > > > > proposal. Nate and David brought it up and shared some insights.
> I'll try
> > > > > > to aggregate them and we can continue the discussion form there.
> > > > > >
> > > > > > Cheers,
> > > > > > Gosh
> > > > > >
> > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > > natebauernfeind@deephaven.io>
> > > > > > wrote:
> > > > > >
> > > > > > > >
> > > > > > > > > > makes it more difficult to bring schema evolution back
> into the
> > > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > > >
> > > > > > > > > Gosh's proposal extends the flatbuffer structures not the
> > > > > protobufs.
> > > > > > > Can
> > > > > > > > > you help me understand how difficult it would be to bring
> the
> > > > > > > `schema_id`
> > > > > > > > > approach to the IPC stream format?
> > > > > > > >
> > > > > > > > I thought we were talking solely about the Flight Protobuf
> > > > > definitions -
> > > > > > > > not the Flatbuffers (and the Google doc at least only talks
> about the
> > > > > > > > Protobufs).
> > > > > > > >
> > > > > > >
> > > > > > > I somehow missed that schema_id is being added to protobuf in
> the
> > > > > document.
> > > > > > > It feels to me that the schema_id is a property that would
> ideally only
> > > > > > > apply to the RecordBatch. I better understand Micah's
> dictionary
> > > > > concerns,
> > > > > > > now, too.
> > > > > > >
> > > > > > > > Side Question: Why isn't the IPC stream format a series of
> the flight
> > > > > > > > > protobufs? It's a real shame that there is no standard way
> to
> > > > > > > > > capture/replay a stream with app_metadata. (Obviously
> ignoring the
> > > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > > >
> > > > > > > > The IPC format was defined long before Flight, and Flight's
> > > > > app_metadata
> > > > > > > > was added after Flight's initial definition. Note an IPC
> message does
> > > > > > > have
> > > > > > > > a provision for key-value metadata, though I think APIs for
> that are
> > > > > not
> > > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and
> despite my
> > > > > comments
> > > > > > > > there perhaps we need to unify or at least consider how
> Flight's
> > > > > > > > app_metadata relates to the IPC message custom_metadata. Also
> > > > > perhaps see
> > > > > > > > ARROW-1059.)
> > > > > > > >
> > > > > > >
> > > > > > > KeyValue unfortunately is string to string. In flatbuffer
> strings are
> > > > > only
> > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is
> opaque
> > > > > bytes.
> > > > > > > The latter is a bit more useful.
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > >
> > > > >
> > >
>


--

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

On Wed, Jul 7, 2021 at 2:53 PM David Li <ap...@lidavidm.me> wrote:
>
> From the Flatbuffers internals doc[1] it appears they are the same: "Strings are simply a vector of bytes, and are always null-terminated."

I see. I took a look at flatbuffers.h, and it appears that changing
this field from string to [byte] would be backward-compatible and
forward-compatible except with code that expects a null terminator.
This is something we could discuss separately if there were enough
interest.

> [1]: https://google.github.io/flatbuffers/flatbuffers_internals.html
>
> -David
>
> On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> > On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <em...@gmail.com> wrote:
> > >
> > > >
> > > > Right, I had wanted to focus the discussion on Flight as I think schema
> > > > evolution or multiplexing streams (more so the latter) is a property of the
> > > > transport and not the stream format itself. If we are leaning towards just
> > > > schema evolution then maybe it makes sense to discuss it for the IPC stream
> > > > format and leverage that in Flight. I'd be interested in what others think.
> > >
> > > I tend to agree, I think stream multiplexing is likely a transport level
> > > issue.  IMO I think schema evolution should be consistent with the IPC
> > > stream format  and flight.
> > >
> > >
> > > > Nate: it may be worth starting a separate discussion about more general
> > > > metadata in the IPC message. I'm not aware of why key-value metadata was
> > > > chosen/if opaque bytes were considered in the past.
> > >
> > >
> > > I think  this was an unfortunate design of the key value metadata in
> > > Schema.fbs, but I don't think I was around when this decision was made.
> >
> > I agree that it's unfortunate that we did not use [ byte ] instead of
> > string for the value in the KeyValue metadata — I think this was more
> > of an oversight than a deliberate choice (e.g. it was not our intent
> > to require binary data to be base64-encoded — this is something that
> > we have to do when encoding binary data in Thrift KeyValue metadata
> > for Parquet, for example). Is the binary representation of [byte]
> > different from string?
> >
> >
> >
> > > Side Question: Why isn't the IPC stream format a series of the flight
> > > > protobufs?
> > >
> > > In addition to what David said, protobufs can't be read directly from a
> > > memory-mapped file (they need decoding).  This was one of the design
> > > considerations of using flatbuffers and IPC Stream/File format.
> > >
> > > I was thinking Micah's comment is more that whatever we do, it should be
> > > > clearly specified and edge cases should be considered, especially if we
> > > > might want to 'backport' this into the stream format later.
> > >
> > >
> > > Yes, for dictionaries we just need to be careful to define semantics and
> > > ensure implementations are validating them with regards to dictionaries.
> > > There likely isn't any need to change current implementations though.
> > >
> > > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org> wrote:
> > >
> > > > Right, I had wanted to focus the discussion on Flight as I think schema
> > > > evolution or multiplexing streams (more so the latter) is a property of the
> > > > transport and not the stream format itself. If we are leaning towards just
> > > > schema evolution then maybe it makes sense to discuss it for the IPC stream
> > > > format and leverage that in Flight. I'd be interested in what others think.
> > > >
> > > > Especially if we are looking at multiplexing streams - I would wonder if
> > > > that's actually better served by making it easier to implement using the
> > > > Flight implementation as it stands (by managing concurrent RPC calls and/or
> > > > performing the union-of-structs encoding trick for you), instead of having
> > > > to change the protocol.
> > > >
> > > > Nate: it may be worth starting a separate discussion about more general
> > > > metadata in the IPC message. I'm not aware of why key-value metadata was
> > > > chosen/if opaque bytes were considered in the past. Off the top of my head
> > > > if it's for on-disk storage and fully application-defined it may make sense
> > > > to store as a separate file alongside the Arrow file (indexed by record
> > > > batch index) where you can take advantage of whatever format is most
> > > > suitable.
> > > >
> > > > -David
> > > >
> > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > > Hi guys,
> > > > >
> > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was to add this
> > > > > feature starting from the IPC(I moved initial write up steps to the
> > > > bottom
> > > > > of the doc). Afterwards David suggested focusing on Flight and that's how
> > > > > we ended up with the protobufs change in the proposal. This being said I
> > > > do
> > > > > think that the place where this should be impemented is a good question
> > > > on
> > > > > its own. Maybe it makes sense to have this kind of a feature in IPC and
> > > > > somehow use it in Flight, maybe not.
> > > > > 2. The point about dictionaries deserves a dedicated section in the
> > > > > proposal. Nate and David brought it up and shared some insights. I'll try
> > > > > to aggregate them and we can continue the discussion form there.
> > > > >
> > > > > Cheers,
> > > > > Gosh
> > > > >
> > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > > natebauernfeind@deephaven.io>
> > > > > wrote:
> > > > >
> > > > > > >
> > > > > > > > > makes it more difficult to bring schema evolution back into the
> > > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > > >
> > > > > > > > Gosh's proposal extends the flatbuffer structures not the
> > > > protobufs.
> > > > > > Can
> > > > > > > > you help me understand how difficult it would be to bring the
> > > > > > `schema_id`
> > > > > > > > approach to the IPC stream format?
> > > > > > >
> > > > > > > I thought we were talking solely about the Flight Protobuf
> > > > definitions -
> > > > > > > not the Flatbuffers (and the Google doc at least only talks about the
> > > > > > > Protobufs).
> > > > > > >
> > > > > >
> > > > > > I somehow missed that schema_id is being added to protobuf in the
> > > > document.
> > > > > > It feels to me that the schema_id is a property that would ideally only
> > > > > > apply to the RecordBatch. I better understand Micah's dictionary
> > > > concerns,
> > > > > > now, too.
> > > > > >
> > > > > > > Side Question: Why isn't the IPC stream format a series of the flight
> > > > > > > > protobufs? It's a real shame that there is no standard way to
> > > > > > > > capture/replay a stream with app_metadata. (Obviously ignoring the
> > > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > > >
> > > > > > > The IPC format was defined long before Flight, and Flight's
> > > > app_metadata
> > > > > > > was added after Flight's initial definition. Note an IPC message does
> > > > > > have
> > > > > > > a provision for key-value metadata, though I think APIs for that are
> > > > not
> > > > > > > fully exposed. (See ARROW-6940:
> > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and despite my
> > > > comments
> > > > > > > there perhaps we need to unify or at least consider how Flight's
> > > > > > > app_metadata relates to the IPC message custom_metadata. Also
> > > > perhaps see
> > > > > > > ARROW-1059.)
> > > > > > >
> > > > > >
> > > > > > KeyValue unfortunately is string to string. In flatbuffer strings are
> > > > only
> > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque
> > > > bytes.
> > > > > > The latter is a bit more useful.
> > > > > >
> > > > > > --
> > > > > >
> > > > >
> > > >
> >

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <ap...@lidavidm.me>]

From the Flatbuffers internals doc[1] it appears they are the same: "Strings are simply a vector of bytes, and are always null-terminated."

[1]: https://google.github.io/flatbuffers/flatbuffers_internals.html

-David

On Wed, Jul 7, 2021, at 05:08, Wes McKinney wrote:
> On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <em...@gmail.com> wrote:
> >
> > >
> > > Right, I had wanted to focus the discussion on Flight as I think schema
> > > evolution or multiplexing streams (more so the latter) is a property of the
> > > transport and not the stream format itself. If we are leaning towards just
> > > schema evolution then maybe it makes sense to discuss it for the IPC stream
> > > format and leverage that in Flight. I'd be interested in what others think.
> >
> > I tend to agree, I think stream multiplexing is likely a transport level
> > issue.  IMO I think schema evolution should be consistent with the IPC
> > stream format  and flight.
> >
> >
> > > Nate: it may be worth starting a separate discussion about more general
> > > metadata in the IPC message. I'm not aware of why key-value metadata was
> > > chosen/if opaque bytes were considered in the past.
> >
> >
> > I think  this was an unfortunate design of the key value metadata in
> > Schema.fbs, but I don't think I was around when this decision was made.
> 
> I agree that it's unfortunate that we did not use [ byte ] instead of
> string for the value in the KeyValue metadata — I think this was more
> of an oversight than a deliberate choice (e.g. it was not our intent
> to require binary data to be base64-encoded — this is something that
> we have to do when encoding binary data in Thrift KeyValue metadata
> for Parquet, for example). Is the binary representation of [byte]
> different from string?
> 
> 
> 
> > Side Question: Why isn't the IPC stream format a series of the flight
> > > protobufs?
> >
> > In addition to what David said, protobufs can't be read directly from a
> > memory-mapped file (they need decoding).  This was one of the design
> > considerations of using flatbuffers and IPC Stream/File format.
> >
> > I was thinking Micah's comment is more that whatever we do, it should be
> > > clearly specified and edge cases should be considered, especially if we
> > > might want to 'backport' this into the stream format later.
> >
> >
> > Yes, for dictionaries we just need to be careful to define semantics and
> > ensure implementations are validating them with regards to dictionaries.
> > There likely isn't any need to change current implementations though.
> >
> > On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org> wrote:
> >
> > > Right, I had wanted to focus the discussion on Flight as I think schema
> > > evolution or multiplexing streams (more so the latter) is a property of the
> > > transport and not the stream format itself. If we are leaning towards just
> > > schema evolution then maybe it makes sense to discuss it for the IPC stream
> > > format and leverage that in Flight. I'd be interested in what others think.
> > >
> > > Especially if we are looking at multiplexing streams - I would wonder if
> > > that's actually better served by making it easier to implement using the
> > > Flight implementation as it stands (by managing concurrent RPC calls and/or
> > > performing the union-of-structs encoding trick for you), instead of having
> > > to change the protocol.
> > >
> > > Nate: it may be worth starting a separate discussion about more general
> > > metadata in the IPC message. I'm not aware of why key-value metadata was
> > > chosen/if opaque bytes were considered in the past. Off the top of my head
> > > if it's for on-disk storage and fully application-defined it may make sense
> > > to store as a separate file alongside the Arrow file (indexed by record
> > > batch index) where you can take advantage of whatever format is most
> > > suitable.
> > >
> > > -David
> > >
> > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > > Hi guys,
> > > >
> > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was to add this
> > > > feature starting from the IPC(I moved initial write up steps to the
> > > bottom
> > > > of the doc). Afterwards David suggested focusing on Flight and that's how
> > > > we ended up with the protobufs change in the proposal. This being said I
> > > do
> > > > think that the place where this should be impemented is a good question
> > > on
> > > > its own. Maybe it makes sense to have this kind of a feature in IPC and
> > > > somehow use it in Flight, maybe not.
> > > > 2. The point about dictionaries deserves a dedicated section in the
> > > > proposal. Nate and David brought it up and shared some insights. I'll try
> > > > to aggregate them and we can continue the discussion form there.
> > > >
> > > > Cheers,
> > > > Gosh
> > > >
> > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > > natebauernfeind@deephaven.io>
> > > > wrote:
> > > >
> > > > > >
> > > > > > > > makes it more difficult to bring schema evolution back into the
> > > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > > >
> > > > > > > Gosh's proposal extends the flatbuffer structures not the
> > > protobufs.
> > > > > Can
> > > > > > > you help me understand how difficult it would be to bring the
> > > > > `schema_id`
> > > > > > > approach to the IPC stream format?
> > > > > >
> > > > > > I thought we were talking solely about the Flight Protobuf
> > > definitions -
> > > > > > not the Flatbuffers (and the Google doc at least only talks about the
> > > > > > Protobufs).
> > > > > >
> > > > >
> > > > > I somehow missed that schema_id is being added to protobuf in the
> > > document.
> > > > > It feels to me that the schema_id is a property that would ideally only
> > > > > apply to the RecordBatch. I better understand Micah's dictionary
> > > concerns,
> > > > > now, too.
> > > > >
> > > > > > Side Question: Why isn't the IPC stream format a series of the flight
> > > > > > > protobufs? It's a real shame that there is no standard way to
> > > > > > > capture/replay a stream with app_metadata. (Obviously ignoring the
> > > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > > >
> > > > > > The IPC format was defined long before Flight, and Flight's
> > > app_metadata
> > > > > > was added after Flight's initial definition. Note an IPC message does
> > > > > have
> > > > > > a provision for key-value metadata, though I think APIs for that are
> > > not
> > > > > > fully exposed. (See ARROW-6940:
> > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and despite my
> > > comments
> > > > > > there perhaps we need to unify or at least consider how Flight's
> > > > > > app_metadata relates to the IPC message custom_metadata. Also
> > > perhaps see
> > > > > > ARROW-1059.)
> > > > > >
> > > > >
> > > > > KeyValue unfortunately is string to string. In flatbuffer strings are
> > > only
> > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque
> > > bytes.
> > > > > The latter is a bit more useful.
> > > > >
> > > > > --
> > > > >
> > > >
> > >
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Wes McKinney <we...@gmail.com>]

On Tue, Jul 6, 2021 at 6:33 PM Micah Kornfield <em...@gmail.com> wrote:
>
> >
> > Right, I had wanted to focus the discussion on Flight as I think schema
> > evolution or multiplexing streams (more so the latter) is a property of the
> > transport and not the stream format itself. If we are leaning towards just
> > schema evolution then maybe it makes sense to discuss it for the IPC stream
> > format and leverage that in Flight. I'd be interested in what others think.
>
> I tend to agree, I think stream multiplexing is likely a transport level
> issue.  IMO I think schema evolution should be consistent with the IPC
> stream format  and flight.
>
>
> > Nate: it may be worth starting a separate discussion about more general
> > metadata in the IPC message. I'm not aware of why key-value metadata was
> > chosen/if opaque bytes were considered in the past.
>
>
> I think  this was an unfortunate design of the key value metadata in
> Schema.fbs, but I don't think I was around when this decision was made.

I agree that it's unfortunate that we did not use [ byte ] instead of
string for the value in the KeyValue metadata — I think this was more
of an oversight than a deliberate choice (e.g. it was not our intent
to require binary data to be base64-encoded — this is something that
we have to do when encoding binary data in Thrift KeyValue metadata
for Parquet, for example). Is the binary representation of [byte]
different from string?



> Side Question: Why isn't the IPC stream format a series of the flight
> > protobufs?
>
> In addition to what David said, protobufs can't be read directly from a
> memory-mapped file (they need decoding).  This was one of the design
> considerations of using flatbuffers and IPC Stream/File format.
>
> I was thinking Micah's comment is more that whatever we do, it should be
> > clearly specified and edge cases should be considered, especially if we
> > might want to 'backport' this into the stream format later.
>
>
> Yes, for dictionaries we just need to be careful to define semantics and
> ensure implementations are validating them with regards to dictionaries.
> There likely isn't any need to change current implementations though.
>
> On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org> wrote:
>
> > Right, I had wanted to focus the discussion on Flight as I think schema
> > evolution or multiplexing streams (more so the latter) is a property of the
> > transport and not the stream format itself. If we are leaning towards just
> > schema evolution then maybe it makes sense to discuss it for the IPC stream
> > format and leverage that in Flight. I'd be interested in what others think.
> >
> > Especially if we are looking at multiplexing streams - I would wonder if
> > that's actually better served by making it easier to implement using the
> > Flight implementation as it stands (by managing concurrent RPC calls and/or
> > performing the union-of-structs encoding trick for you), instead of having
> > to change the protocol.
> >
> > Nate: it may be worth starting a separate discussion about more general
> > metadata in the IPC message. I'm not aware of why key-value metadata was
> > chosen/if opaque bytes were considered in the past. Off the top of my head
> > if it's for on-disk storage and fully application-defined it may make sense
> > to store as a separate file alongside the Arrow file (indexed by record
> > batch index) where you can take advantage of whatever format is most
> > suitable.
> >
> > -David
> >
> > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > > Hi guys,
> > >
> > > 1. Regarding IPC vs Flight: in fact my initial suggestion was to add this
> > > feature starting from the IPC(I moved initial write up steps to the
> > bottom
> > > of the doc). Afterwards David suggested focusing on Flight and that's how
> > > we ended up with the protobufs change in the proposal. This being said I
> > do
> > > think that the place where this should be impemented is a good question
> > on
> > > its own. Maybe it makes sense to have this kind of a feature in IPC and
> > > somehow use it in Flight, maybe not.
> > > 2. The point about dictionaries deserves a dedicated section in the
> > > proposal. Nate and David brought it up and shared some insights. I'll try
> > > to aggregate them and we can continue the discussion form there.
> > >
> > > Cheers,
> > > Gosh
> > >
> > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> > natebauernfeind@deephaven.io>
> > > wrote:
> > >
> > > > >
> > > > > > > makes it more difficult to bring schema evolution back into the
> > > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > > >
> > > > > > Gosh's proposal extends the flatbuffer structures not the
> > protobufs.
> > > > Can
> > > > > > you help me understand how difficult it would be to bring the
> > > > `schema_id`
> > > > > > approach to the IPC stream format?
> > > > >
> > > > > I thought we were talking solely about the Flight Protobuf
> > definitions -
> > > > > not the Flatbuffers (and the Google doc at least only talks about the
> > > > > Protobufs).
> > > > >
> > > >
> > > > I somehow missed that schema_id is being added to protobuf in the
> > document.
> > > > It feels to me that the schema_id is a property that would ideally only
> > > > apply to the RecordBatch. I better understand Micah's dictionary
> > concerns,
> > > > now, too.
> > > >
> > > > > Side Question: Why isn't the IPC stream format a series of the flight
> > > > > > protobufs? It's a real shame that there is no standard way to
> > > > > > capture/replay a stream with app_metadata. (Obviously ignoring the
> > > > > > annoyances around protobuf wrapping flatbuffers.)
> > > > >
> > > > > The IPC format was defined long before Flight, and Flight's
> > app_metadata
> > > > > was added after Flight's initial definition. Note an IPC message does
> > > > have
> > > > > a provision for key-value metadata, though I think APIs for that are
> > not
> > > > > fully exposed. (See ARROW-6940:
> > > > > https://issues.apache.org/jira/browse/ARROW-6940 and despite my
> > comments
> > > > > there perhaps we need to unify or at least consider how Flight's
> > > > > app_metadata relates to the IPC message custom_metadata. Also
> > perhaps see
> > > > > ARROW-1059.)
> > > > >
> > > >
> > > > KeyValue unfortunately is string to string. In flatbuffer strings are
> > only
> > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque
> > bytes.
> > > > The latter is a bit more useful.
> > > >
> > > > --
> > > >
> > >
> >

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

>
> Right, I had wanted to focus the discussion on Flight as I think schema
> evolution or multiplexing streams (more so the latter) is a property of the
> transport and not the stream format itself. If we are leaning towards just
> schema evolution then maybe it makes sense to discuss it for the IPC stream
> format and leverage that in Flight. I'd be interested in what others think.

I tend to agree, I think stream multiplexing is likely a transport level
issue.  IMO I think schema evolution should be consistent with the IPC
stream format  and flight.


> Nate: it may be worth starting a separate discussion about more general
> metadata in the IPC message. I'm not aware of why key-value metadata was
> chosen/if opaque bytes were considered in the past.


I think  this was an unfortunate design of the key value metadata in
Schema.fbs, but I don't think I was around when this decision was made.

Side Question: Why isn't the IPC stream format a series of the flight
> protobufs?

In addition to what David said, protobufs can't be read directly from a
memory-mapped file (they need decoding).  This was one of the design
considerations of using flatbuffers and IPC Stream/File format.

I was thinking Micah's comment is more that whatever we do, it should be
> clearly specified and edge cases should be considered, especially if we
> might want to 'backport' this into the stream format later.


Yes, for dictionaries we just need to be careful to define semantics and
ensure implementations are validating them with regards to dictionaries.
There likely isn't any need to change current implementations though.

On Mon, Jun 28, 2021 at 1:25 PM David Li <li...@apache.org> wrote:

> Right, I had wanted to focus the discussion on Flight as I think schema
> evolution or multiplexing streams (more so the latter) is a property of the
> transport and not the stream format itself. If we are leaning towards just
> schema evolution then maybe it makes sense to discuss it for the IPC stream
> format and leverage that in Flight. I'd be interested in what others think.
>
> Especially if we are looking at multiplexing streams - I would wonder if
> that's actually better served by making it easier to implement using the
> Flight implementation as it stands (by managing concurrent RPC calls and/or
> performing the union-of-structs encoding trick for you), instead of having
> to change the protocol.
>
> Nate: it may be worth starting a separate discussion about more general
> metadata in the IPC message. I'm not aware of why key-value metadata was
> chosen/if opaque bytes were considered in the past. Off the top of my head
> if it's for on-disk storage and fully application-defined it may make sense
> to store as a separate file alongside the Arrow file (indexed by record
> batch index) where you can take advantage of whatever format is most
> suitable.
>
> -David
>
> On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> > Hi guys,
> >
> > 1. Regarding IPC vs Flight: in fact my initial suggestion was to add this
> > feature starting from the IPC(I moved initial write up steps to the
> bottom
> > of the doc). Afterwards David suggested focusing on Flight and that's how
> > we ended up with the protobufs change in the proposal. This being said I
> do
> > think that the place where this should be impemented is a good question
> on
> > its own. Maybe it makes sense to have this kind of a feature in IPC and
> > somehow use it in Flight, maybe not.
> > 2. The point about dictionaries deserves a dedicated section in the
> > proposal. Nate and David brought it up and shared some insights. I'll try
> > to aggregate them and we can continue the discussion form there.
> >
> > Cheers,
> > Gosh
> >
> > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <
> natebauernfeind@deephaven.io>
> > wrote:
> >
> > > >
> > > > > > makes it more difficult to bring schema evolution back into the
> > > > > > IPC Stream format (i.e. it would live only in flight)
> > > > >
> > > > > Gosh's proposal extends the flatbuffer structures not the
> protobufs.
> > > Can
> > > > > you help me understand how difficult it would be to bring the
> > > `schema_id`
> > > > > approach to the IPC stream format?
> > > >
> > > > I thought we were talking solely about the Flight Protobuf
> definitions -
> > > > not the Flatbuffers (and the Google doc at least only talks about the
> > > > Protobufs).
> > > >
> > >
> > > I somehow missed that schema_id is being added to protobuf in the
> document.
> > > It feels to me that the schema_id is a property that would ideally only
> > > apply to the RecordBatch. I better understand Micah's dictionary
> concerns,
> > > now, too.
> > >
> > > > Side Question: Why isn't the IPC stream format a series of the flight
> > > > > protobufs? It's a real shame that there is no standard way to
> > > > > capture/replay a stream with app_metadata. (Obviously ignoring the
> > > > > annoyances around protobuf wrapping flatbuffers.)
> > > >
> > > > The IPC format was defined long before Flight, and Flight's
> app_metadata
> > > > was added after Flight's initial definition. Note an IPC message does
> > > have
> > > > a provision for key-value metadata, though I think APIs for that are
> not
> > > > fully exposed. (See ARROW-6940:
> > > > https://issues.apache.org/jira/browse/ARROW-6940 and despite my
> comments
> > > > there perhaps we need to unify or at least consider how Flight's
> > > > app_metadata relates to the IPC message custom_metadata. Also
> perhaps see
> > > > ARROW-1059.)
> > > >
> > >
> > > KeyValue unfortunately is string to string. In flatbuffer strings are
> only
> > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque
> bytes.
> > > The latter is a bit more useful.
> > >
> > > --
> > >
> >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Right, I had wanted to focus the discussion on Flight as I think schema evolution or multiplexing streams (more so the latter) is a property of the transport and not the stream format itself. If we are leaning towards just schema evolution then maybe it makes sense to discuss it for the IPC stream format and leverage that in Flight. I'd be interested in what others think.

Especially if we are looking at multiplexing streams - I would wonder if that's actually better served by making it easier to implement using the Flight implementation as it stands (by managing concurrent RPC calls and/or performing the union-of-structs encoding trick for you), instead of having to change the protocol.

Nate: it may be worth starting a separate discussion about more general metadata in the IPC message. I'm not aware of why key-value metadata was chosen/if opaque bytes were considered in the past. Off the top of my head if it's for on-disk storage and fully application-defined it may make sense to store as a separate file alongside the Arrow file (indexed by record batch index) where you can take advantage of whatever format is most suitable.

-David

On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote:
> Hi guys,
> 
> 1. Regarding IPC vs Flight: in fact my initial suggestion was to add this
> feature starting from the IPC(I moved initial write up steps to the bottom
> of the doc). Afterwards David suggested focusing on Flight and that's how
> we ended up with the protobufs change in the proposal. This being said I do
> think that the place where this should be impemented is a good question on
> its own. Maybe it makes sense to have this kind of a feature in IPC and
> somehow use it in Flight, maybe not.
> 2. The point about dictionaries deserves a dedicated section in the
> proposal. Nate and David brought it up and shared some insights. I'll try
> to aggregate them and we can continue the discussion form there.
> 
> Cheers,
> Gosh
> 
> On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <na...@deephaven.io>
> wrote:
> 
> > >
> > > > > makes it more difficult to bring schema evolution back into the
> > > > > IPC Stream format (i.e. it would live only in flight)
> > > >
> > > > Gosh's proposal extends the flatbuffer structures not the protobufs.
> > Can
> > > > you help me understand how difficult it would be to bring the
> > `schema_id`
> > > > approach to the IPC stream format?
> > >
> > > I thought we were talking solely about the Flight Protobuf definitions -
> > > not the Flatbuffers (and the Google doc at least only talks about the
> > > Protobufs).
> > >
> >
> > I somehow missed that schema_id is being added to protobuf in the document.
> > It feels to me that the schema_id is a property that would ideally only
> > apply to the RecordBatch. I better understand Micah's dictionary concerns,
> > now, too.
> >
> > > Side Question: Why isn't the IPC stream format a series of the flight
> > > > protobufs? It's a real shame that there is no standard way to
> > > > capture/replay a stream with app_metadata. (Obviously ignoring the
> > > > annoyances around protobuf wrapping flatbuffers.)
> > >
> > > The IPC format was defined long before Flight, and Flight's app_metadata
> > > was added after Flight's initial definition. Note an IPC message does
> > have
> > > a provision for key-value metadata, though I think APIs for that are not
> > > fully exposed. (See ARROW-6940:
> > > https://issues.apache.org/jira/browse/ARROW-6940 and despite my comments
> > > there perhaps we need to unify or at least consider how Flight's
> > > app_metadata relates to the IPC message custom_metadata. Also perhaps see
> > > ARROW-1059.)
> > >
> >
> > KeyValue unfortunately is string to string. In flatbuffer strings are only
> > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque bytes.
> > The latter is a bit more useful.
> >
> > --
> >
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi guys,

1. Regarding IPC vs Flight: in fact my initial suggestion was to add this
feature starting from the IPC(I moved initial write up steps to the bottom
of the doc). Afterwards David suggested focusing on Flight and that's how
we ended up with the protobufs change in the proposal. This being said I do
think that the place where this should be impemented is a good question on
its own. Maybe it makes sense to have this kind of a feature in IPC and
somehow use it in Flight, maybe not.
2. The point about dictionaries deserves a dedicated section in the
proposal. Nate and David brought it up and shared some insights. I'll try
to aggregate them and we can continue the discussion form there.

Cheers,
Gosh

On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, <na...@deephaven.io>
wrote:

> >
> > > > makes it more difficult to bring schema evolution back into the
> > > > IPC Stream format (i.e. it would live only in flight)
> > >
> > > Gosh's proposal extends the flatbuffer structures not the protobufs.
> Can
> > > you help me understand how difficult it would be to bring the
> `schema_id`
> > > approach to the IPC stream format?
> >
> > I thought we were talking solely about the Flight Protobuf definitions -
> > not the Flatbuffers (and the Google doc at least only talks about the
> > Protobufs).
> >
>
> I somehow missed that schema_id is being added to protobuf in the document.
> It feels to me that the schema_id is a property that would ideally only
> apply to the RecordBatch. I better understand Micah's dictionary concerns,
> now, too.
>
> > Side Question: Why isn't the IPC stream format a series of the flight
> > > protobufs? It's a real shame that there is no standard way to
> > > capture/replay a stream with app_metadata. (Obviously ignoring the
> > > annoyances around protobuf wrapping flatbuffers.)
> >
> > The IPC format was defined long before Flight, and Flight's app_metadata
> > was added after Flight's initial definition. Note an IPC message does
> have
> > a provision for key-value metadata, though I think APIs for that are not
> > fully exposed. (See ARROW-6940:
> > https://issues.apache.org/jira/browse/ARROW-6940 and despite my comments
> > there perhaps we need to unify or at least consider how Flight's
> > app_metadata relates to the IPC message custom_metadata. Also perhaps see
> > ARROW-1059.)
> >
>
> KeyValue unfortunately is string to string. In flatbuffer strings are only
> UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque bytes.
> The latter is a bit more useful.
>
> --
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Nate Bauernfeind <na...@deephaven.io>]

>
> > > makes it more difficult to bring schema evolution back into the
> > > IPC Stream format (i.e. it would live only in flight)
> >
> > Gosh's proposal extends the flatbuffer structures not the protobufs. Can
> > you help me understand how difficult it would be to bring the `schema_id`
> > approach to the IPC stream format?
>
> I thought we were talking solely about the Flight Protobuf definitions -
> not the Flatbuffers (and the Google doc at least only talks about the
> Protobufs).
>

I somehow missed that schema_id is being added to protobuf in the document.
It feels to me that the schema_id is a property that would ideally only
apply to the RecordBatch. I better understand Micah's dictionary concerns,
now, too.

> Side Question: Why isn't the IPC stream format a series of the flight
> > protobufs? It's a real shame that there is no standard way to
> > capture/replay a stream with app_metadata. (Obviously ignoring the
> > annoyances around protobuf wrapping flatbuffers.)
>
> The IPC format was defined long before Flight, and Flight's app_metadata
> was added after Flight's initial definition. Note an IPC message does have
> a provision for key-value metadata, though I think APIs for that are not
> fully exposed. (See ARROW-6940:
> https://issues.apache.org/jira/browse/ARROW-6940 and despite my comments
> there perhaps we need to unify or at least consider how Flight's
> app_metadata relates to the IPC message custom_metadata. Also perhaps see
> ARROW-1059.)
>

KeyValue unfortunately is string to string. In flatbuffer strings are only
UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque bytes.
The latter is a bit more useful.

--

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Replies inline:

On Fri, Jun 25, 2021, at 20:18, Nate Bauernfeind wrote:
> > makes it more difficult to bring schema evolution back into the
> > IPC Stream format (i.e. it would live only in flight)
> 
> Gosh's proposal extends the flatbuffer structures not the protobufs. Can
> you help me understand how difficult it would be to bring the `schema_id`
> approach to the IPC stream format?

I thought we were talking solely about the Flight Protobuf definitions - not the Flatbuffers (and the Google doc at least only talks about the Protobufs).

I do agree with Micah that we should clarify the exact goals. It really does sound like there are two entirely separate use cases here. Furthermore it might be helpful to demonstrate (with code) how complex it would (or wouldn't) be to use separate RPC calls for the switch-between-known-schemas case since that would avoid pushing down complexity into Flight itself. I could sketch this if it might be useful.

> 
> Side Question: Why isn't the IPC stream format a series of the flight
> protobufs? It's a real shame that there is no standard way to
> capture/replay a stream with app_metadata. (Obviously ignoring the
> annoyances around protobuf wrapping flatbuffers.)

The IPC format was defined long before Flight, and Flight's app_metadata was added after Flight's initial definition. Note an IPC message does have a provision for key-value metadata, though I think APIs for that are not fully exposed. (See ARROW-6940: https://issues.apache.org/jira/browse/ARROW-6940 and despite my comments there perhaps we need to unify or at least consider how Flight's app_metadata relates to the IPC message custom_metadata. Also perhaps see ARROW-1059.)

> 
> > In both cases there are also potentially thorny issues related to
> > dictionary usages that we will need to resolve.
> 
> I hope changing how dictionaries work can be avoided. There are some nice
> benefits to sharing a dictionary across schema evolution (esp. for frequent
> switches between a few schemas). Don't forget, we already have the ability
> to clear / overwrite existing dictionaries. It might be nice to leave the
> behavior as-is.

I was thinking Micah's comment is more that whatever we do, it should be clearly specified and edge cases should be considered, especially if we might want to 'backport' this into the stream format later.

> 
> On Fri, Jun 25, 2021 at 4:07 PM Micah Kornfield <em...@gmail.com>
> wrote:
> 
> > Sorry for the second reply:
> >
> >
> >>    1. In our case we do expect relatively frequent changes in the schema
> >>    of the batch being sent out. I don't see that pattern changing in the mid
> >>    term for a good reason. However long term maybe it will be possible to
> >>    leverage separate RPC calls. I left some description in the comments thread
> >>    here
> >>    <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
> >>    .
> >>
> >> I think we should clarify between two types of changes:
> > 1.  Completely new  unique schema needed.
> > 2.  Relatively frequent switches between a few known schemas.
> >
> > On Fri, Jun 25, 2021 at 2:58 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >>
> >>>    1. Re complexity of "one schema at a time" vs "schema id based": i
> >>>    think they are not much different, right? In fact the second one is more of
> >>>    an optimization to the first one which is beneficial to us. Anyways even
> >>>    with the first approach you need to add some logic of schema change. Adding
> >>>    the schema identification is not that complex at first glance.
> >>>
> >>> The second approach requires more state being maintained on the client
> >> side and makes it more difficult to bring schema evolution back into the
> >> IPC Stream format (i.e. it would live only in flight) without additional
> >> changes.
> >>
> >> In both cases there are also potentially thorny issues related to
> >> dictionary usages that we will need to resolve.  I haven't thought deeply
> >> about it but it seems that these might be more complex on a multiplexed
> >> channel.
> >>
> >>
> >> On Fri, Jun 25, 2021 at 11:52 AM Gosh Arzumanyan <go...@gmail.com>
> >> wrote:
> >>
> >>> Hi Micah,
> >>>
> >>> Sure, let me do it here:
> >>>
> >>>    1. In our case we do expect relatively frequent changes in the
> >>>    schema of the batch being sent out. I don't see that pattern changing in
> >>>    the mid term for a good reason. However long term maybe it will be possible
> >>>    to leverage separate RPC calls. I left some description in the comments
> >>>    thread here
> >>>    <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
> >>>    .
> >>>    2. Regarding the union of structs trick: while it is a good
> >>>    workaround for most of the cases as of now there are also some aspects
> >>>    which deserve consideration:
> >>>       1. There is a  space overhead
> >>>       2. There is an additional glue code to make it work
> >>>       3. Considering that this solution is bubbling up frequently in
> >>>       similar contexts, seems like users might benefit if it  was "natively"
> >>>       supported and they could just focus on populating the schemas they really
> >>>       need to.
> >>>    3. Re complexity of "one schema at a time" vs "schema id based": i
> >>>    think they are not much different, right? In fact the second one is more of
> >>>    an optimization to the first one which is beneficial to us. Anyways even
> >>>    with the first approach you need to add some logic of schema change. Adding
> >>>    the schema identification is not that complex at first glance.
> >>>
> >>> Cheers,
> >>> Gosh
> >>>
> >>> On Fri, Jun 25, 2021 at 6:25 PM Micah Kornfield <em...@gmail.com>
> >>> wrote:
> >>>
> >>>> >
> >>>> > 1. It seems like renaming stream_id to schema_id and delegating
> >>>> "logical
> >>>> > stream" distinction to app_metadata mitigates the "multiplexing" point
> >>>> > while at the same time it gives enough flexibility to address both
> >>>> Nate's
> >>>> > and our use cases.
> >>>>
> >>>>
> >>>> I don't think this is the case.  It seems that having no additional
> >>>> fields
> >>>> added and then sending a new schema when necessary combined with Union
> >>>> of
> >>>> Structs would solve most use-cases.  The main downside could be
> >>>> potential
> >>>> performance implications if the schema is changing frequently.  Gosh,
> >>>> could
> >>>> you address why this wouldn't be sufficient (either here or on the doc).
> >>>>
> >>>> Thanks,
> >>>> -Micah
> >>>>
> >>>> On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan <go...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Hi guys,
> >>>> >
> >>>> > Thanks for sharing your insights/concerns! I also left some comments
> >>>> based
> >>>> > on the discussion we had. Briefly:
> >>>> >
> >>>> > 1. It seems like renaming stream_id to schema_id and delegating
> >>>> "logical
> >>>> > stream" distinction to app_metadata mitigates the "multiplexing" point
> >>>> > while at the same time it gives enough flexibility to address both
> >>>> Nate's
> >>>> > and our use cases.
> >>>> > 2. To David's point about other transports: in fact currently we are
> >>>> using
> >>>> > other transports(aside from gRPC) so we don't wanna depend on only
> >>>> gRPC
> >>>> > features.
> >>>> >
> >>>> > Cheers,
> >>>> > Gosh
> >>>> >
> >>>> > On Wed, Jun 23, 2021 at 10:40 PM David Li <li...@apache.org>
> >>>> wrote:
> >>>> >
> >>>> > > Thanks for chiming in - I've replied in the doc. Scoping it to just
> >>>> > schema
> >>>> > > evolution would be preferable, but I'm not sure if Gosh's usecase
> >>>> > requires
> >>>> > > more flexibility than that or not.
> >>>> > >
> >>>> > > Again, though, given that 1) gRPC recycles a connection, so repeated
> >>>> > calls
> >>>> > > aren't necessarily expensive and 2) encoding tricks like
> >>>> > union-of-structs,
> >>>> > > any solution needs to be weighed against those/we should make sure
> >>>> to
> >>>> > > document why they aren't sufficient. (For instance, 1) is hampered
> >>>> by the
> >>>> > > use of L7 load balancers and/or client-side load balancing policies
> >>>> in
> >>>> > gRPC
> >>>> > > and assumes statefulness which is undesirable in general. There's
> >>>> also
> >>>> > the
> >>>> > > eventual desire to have a transport besides gRPC someday.)
> >>>> > >
> >>>> > > -David
> >>>> > >
> >>>> > > On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
> >>>> > >
> >>>> > > Thanks for writing this up! I added a few general comments, but
> >>>> have a
> >>>> > > question on the approach because it's not quite what I was
> >>>> expecting.
> >>>> > >
> >>>> > > I am slightly concerned that the proposal looks more like support
> >>>> for
> >>>> > > "multiplexing" IPC streams into a single RPC stream rather than
> >>>> support
> >>>> > for
> >>>> > > a changing Schema of an otherwise consistently logical stream. gRPC
> >>>> > already
> >>>> > > does a good job decoupling RPC streams from one another. I feel that
> >>>> > > throwing that idea into the IPC stream increases client-library
> >>>> > > implementation cost by quite a lot.
> >>>> > >
> >>>> > > Why is it not good enough to replace the Schema when we see a
> >>>> duplicate?
> >>>> > > This is undoubtedly less work across all client implementations.
> >>>> > >
> >>>> > > The benefit I see is that you might have two schemas that you swap
> >>>> > between
> >>>> > > frequently then you can indicate with a single integer. If that's
> >>>> what
> >>>> > you
> >>>> > > want to support I would rather think of them as `schema_id` instead
> >>>> of
> >>>> > > `stream_id` and not give this impression that multiplexing is a
> >>>> goal. As
> >>>> > > you have proposed, it seems that the "done writing for a stream"
> >>>> needs a
> >>>> > > callback notifying the user receiving the stream that a logical
> >>>> subset of
> >>>> > > the flight is complete. Alternatively, if they aren't independent
> >>>> streams
> >>>> > > (to the end-user), we could tell the Arrow layer that a particular
> >>>> schema
> >>>> > > is no longer needed without also needing to communicate further
> >>>> > downstream.
> >>>> > >
> >>>> > > On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org>
> >>>> wrote:
> >>>> > >
> >>>> > > > Ah to be clear, the API is indeed inconsistent - DoExchange was
> >>>> added
> >>>> > > some
> >>>> > > > time later (and by its nature returning a FlightDataStream would
> >>>> not
> >>>> > have
> >>>> > > > been possible, since it's meant to be able to interleave
> >>>> > > reading/writing).
> >>>> > > > But really, DoGet is indeed the odd one out in the C++ API and it
> >>>> may
> >>>> > be
> >>>> > > > worth correcting. You could also perhaps imagine making a
> >>>> > > FlightDataStream
> >>>> > > > implementation that accepts a closure and provides it a fake
> >>>> writer, if
> >>>> > > the
> >>>> > > > API mismatch is hard to work with...
> >>>> > > >
> >>>> > > > That said: this has some benefits, e.g. for a Python service that
> >>>> > returns
> >>>> > > > a Table, that means data can be fed into gRPC entirely in C++
> >>>> without
> >>>> > > > having to bounce into Python for each chunk.
> >>>> > > >
> >>>> > > > Best,
> >>>> > > > David
> >>>> > > >
> >>>> > > > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
> >>>> > > > > Hi David,
> >>>> > > > >
> >>>> > > > > Got you. In fact I was looking at this more from the point of
> >>>> view of
> >>>> > > > consistency of the API in terms of "inputs" and thought
> >>>> DoExchange is
> >>>> > > kind
> >>>> > > > of a DoGet+ so might make sense to have the same classes being
> >>>> utilized
> >>>> > > in
> >>>> > > > both places. But again, I might be missing something and I get the
> >>>> > point
> >>>> > > > about breaking change.
> >>>> > > > >
> >>>> > > > > Cheers,
> >>>> > > > > Gosh
> >>>> > > > >
> >>>> > > > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org>
> >>>> > wrote:
> >>>> > > > >> __
> >>>> > > > >> It's mostly a quirk of implementation (and just for
> >>>> clarification,
> >>>> > > > they're all nearly identical on the format/protocol level).
> >>>> > > > >>
> >>>> > > > >> DoGet is conceptualized as your application returning a
> >>>> readable
> >>>> > > stream
> >>>> > > > of batches, instead of your application imperatively writing
> >>>> batches to
> >>>> > > the
> >>>> > > > client. (This is different than how Flight is implemented in
> >>>> Java.) You
> >>>> > > > would normally not implement FlightDataStream - you would return a
> >>>> > > > RecordBatchStream.
> >>>> > > > >>
> >>>> > > > >> DoGet could not have FlightMessageWriter as a return type as
> >>>> that
> >>>> > > > wouldn't make sense, but it could accept an instance of that as a
> >>>> > > parameter
> >>>> > > > instead, much like DoExchange. That would be a breaking change.
> >>>> > > > >>
> >>>> > > > >> Best,
> >>>> > > > >> David
> >>>> > > > >>
> >>>> > > > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
> >>>> > > > >>> Hi David,
> >>>> > > > >>>
> >>>> > > > >>> Going through the ArrowFlight API: got confused a bit on
> >>>> DoGet and
> >>>> > > > >>> DoPut/DoExachange apis: the former one expects
> >>>> FlightDataStream
> >>>> > which
> >>>> > > > talks
> >>>> > > > >>> in already serialized message terms while the latter to
> >>>> > > > >>> accept FlightMessageReader/Writer which expect the user to
> >>>> pass in
> >>>> > > > >>> RecordBatches etc. Is there any reason why the DoGet can't
> >>>> have
> >>>> > > > >>> FlightMessageWriter as a return type?
> >>>> > > > >>>
> >>>> > > > >>> Cheers,
> >>>> > > > >>> Gosh
> >>>> > > > >>>
> >>>> > > > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <
> >>>> gosharz@gmail.com
> >>>> > >
> >>>> > > > wrote:
> >>>> > > > >>>
> >>>> > > > >>> > Thanks David!
> >>>> > > > >>> >
> >>>> > > > >>> > I also responded/added more suggestions/questions to the
> >>>> doc. I
> >>>> > > > think it
> >>>> > > > >>> > makes sense to have two sections: one purely protocol
> >>>> oriented
> >>>> > and
> >>>> > > > second
> >>>> > > > >>> > API oriented(examples in c++ or in any other language
> >>>> should make
> >>>> > > > the idea
> >>>> > > > >>> > easier to digest).
> >>>> > > > >>> >
> >>>> > > > >>> > Thanks for the reference too!
> >>>> > > > >>> >
> >>>> > > > >>> > Cheers,
> >>>> > > > >>> > Gosh
> >>>> > > > >>> >
> >>>> > > > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <
> >>>> lidavidm@apache.org>
> >>>> > > > wrote:
> >>>> > > > >>> >
> >>>> > > > >>> >> Thanks! I've left some initial comments/suggestions to
> >>>> expand it
> >>>> > > in
> >>>> > > > terms
> >>>> > > > >>> >> of the format definitions and not the C++ APIs.
> >>>> > > > >>> >>
> >>>> > > > >>> >> I'll also note something like this was proposed a long
> >>>> time ago
> >>>> > -
> >>>> > > > there's
> >>>> > > > >>> >> not very much discussion about it there but for reference:
> >>>> > > > >>> >>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> >>>> > > > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to
> >>>> Flight:
> >>>> > > > >>> >> "DoBidirectional"' from 2019-2020). It might be good to
> >>>> address
> >>>> > > why
> >>>> > > > the
> >>>> > > > >>> >> proposed workaround there (union-of-structs) is
> >>>> insufficient for
> >>>> > > > the use
> >>>> > > > >>> >> cases here (and in FlightSQL).
> >>>> > > > >>> >>
> >>>> > > > >>> >> -David
> >>>> > > > >>> >>
> >>>> > > > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> >>>> > > > >>> >> > Ah sorry, comments should work now.
> >>>> > > > >>> >> >
> >>>> > > > >>> >> > Cheers,
> >>>> > > > >>> >> > Gosh
> >>>> > > > >>> >> >
> >>>> > > > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <
> >>>> lidavidm@apache.org
> >>>> > > > <mailto:
> >>>> > > > >>> >> lidavidm%40apache.org>> wrote:
> >>>> > > > >>> >> >
> >>>> > > > >>> >> > > Thanks! Will give it a look.
> >>>> > > > >>> >> > >
> >>>> > > > >>> >> > > Would you mind opening it up for comments?
> >>>> > > > >>> >> > >
> >>>> > > > >>> >> > > -David
> >>>> > > > >>> >> > >
> >>>> > > > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <
> >>>> gosharz@gmail.com
> >>>> > > > <mailto:
> >>>> > > > >>> >> gosharz%40gmail.com>> wrote:
> >>>> > > > >>> >> > > > Hi folks,
> >>>> > > > >>> >> > > >
> >>>> > > > >>> >> > > > Started putting some thoughts together here:
> >>>> > > > >>> >> > > >
> >>>> > > > >>> >> > >
> >>>> > > > >>> >>
> >>>> > > >
> >>>> > >
> >>>> >
> >>>> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> >>>> > > > >>> >> > > > Any feedback is welcome!
> >>>> > > > >>> >> > > >
> >>>> > > > >>> >> > > > Cheers,
> >>>> > > > >>> >> > > > Gosh
> >>>> > > > >>> >> > > >
> >>>> > > > >>> >> > >
> >>>> > > > >>> >> >
> >>>> > > > >>> >>
> >>>> > > > >>> >
> >>>> > > > >>>
> >>>> > > > >>
> >>>> > > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> 
> --
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Nate Bauernfeind <na...@deephaven.io>]

> makes it more difficult to bring schema evolution back into the
> IPC Stream format (i.e. it would live only in flight)

Gosh's proposal extends the flatbuffer structures not the protobufs. Can
you help me understand how difficult it would be to bring the `schema_id`
approach to the IPC stream format?

Side Question: Why isn't the IPC stream format a series of the flight
protobufs? It's a real shame that there is no standard way to
capture/replay a stream with app_metadata. (Obviously ignoring the
annoyances around protobuf wrapping flatbuffers.)

> In both cases there are also potentially thorny issues related to
> dictionary usages that we will need to resolve.

I hope changing how dictionaries work can be avoided. There are some nice
benefits to sharing a dictionary across schema evolution (esp. for frequent
switches between a few schemas). Don't forget, we already have the ability
to clear / overwrite existing dictionaries. It might be nice to leave the
behavior as-is.

On Fri, Jun 25, 2021 at 4:07 PM Micah Kornfield <em...@gmail.com>
wrote:

> Sorry for the second reply:
>
>
>>    1. In our case we do expect relatively frequent changes in the schema
>>    of the batch being sent out. I don't see that pattern changing in the mid
>>    term for a good reason. However long term maybe it will be possible to
>>    leverage separate RPC calls. I left some description in the comments thread
>>    here
>>    <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
>>    .
>>
>> I think we should clarify between two types of changes:
> 1.  Completely new  unique schema needed.
> 2.  Relatively frequent switches between a few known schemas.
>
> On Fri, Jun 25, 2021 at 2:58 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>>
>>>    1. Re complexity of "one schema at a time" vs "schema id based": i
>>>    think they are not much different, right? In fact the second one is more of
>>>    an optimization to the first one which is beneficial to us. Anyways even
>>>    with the first approach you need to add some logic of schema change. Adding
>>>    the schema identification is not that complex at first glance.
>>>
>>> The second approach requires more state being maintained on the client
>> side and makes it more difficult to bring schema evolution back into the
>> IPC Stream format (i.e. it would live only in flight) without additional
>> changes.
>>
>> In both cases there are also potentially thorny issues related to
>> dictionary usages that we will need to resolve.  I haven't thought deeply
>> about it but it seems that these might be more complex on a multiplexed
>> channel.
>>
>>
>> On Fri, Jun 25, 2021 at 11:52 AM Gosh Arzumanyan <go...@gmail.com>
>> wrote:
>>
>>> Hi Micah,
>>>
>>> Sure, let me do it here:
>>>
>>>    1. In our case we do expect relatively frequent changes in the
>>>    schema of the batch being sent out. I don't see that pattern changing in
>>>    the mid term for a good reason. However long term maybe it will be possible
>>>    to leverage separate RPC calls. I left some description in the comments
>>>    thread here
>>>    <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
>>>    .
>>>    2. Regarding the union of structs trick: while it is a good
>>>    workaround for most of the cases as of now there are also some aspects
>>>    which deserve consideration:
>>>       1. There is a  space overhead
>>>       2. There is an additional glue code to make it work
>>>       3. Considering that this solution is bubbling up frequently in
>>>       similar contexts, seems like users might benefit if it  was "natively"
>>>       supported and they could just focus on populating the schemas they really
>>>       need to.
>>>    3. Re complexity of "one schema at a time" vs "schema id based": i
>>>    think they are not much different, right? In fact the second one is more of
>>>    an optimization to the first one which is beneficial to us. Anyways even
>>>    with the first approach you need to add some logic of schema change. Adding
>>>    the schema identification is not that complex at first glance.
>>>
>>> Cheers,
>>> Gosh
>>>
>>> On Fri, Jun 25, 2021 at 6:25 PM Micah Kornfield <em...@gmail.com>
>>> wrote:
>>>
>>>> >
>>>> > 1. It seems like renaming stream_id to schema_id and delegating
>>>> "logical
>>>> > stream" distinction to app_metadata mitigates the "multiplexing" point
>>>> > while at the same time it gives enough flexibility to address both
>>>> Nate's
>>>> > and our use cases.
>>>>
>>>>
>>>> I don't think this is the case.  It seems that having no additional
>>>> fields
>>>> added and then sending a new schema when necessary combined with Union
>>>> of
>>>> Structs would solve most use-cases.  The main downside could be
>>>> potential
>>>> performance implications if the schema is changing frequently.  Gosh,
>>>> could
>>>> you address why this wouldn't be sufficient (either here or on the doc).
>>>>
>>>> Thanks,
>>>> -Micah
>>>>
>>>> On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan <go...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi guys,
>>>> >
>>>> > Thanks for sharing your insights/concerns! I also left some comments
>>>> based
>>>> > on the discussion we had. Briefly:
>>>> >
>>>> > 1. It seems like renaming stream_id to schema_id and delegating
>>>> "logical
>>>> > stream" distinction to app_metadata mitigates the "multiplexing" point
>>>> > while at the same time it gives enough flexibility to address both
>>>> Nate's
>>>> > and our use cases.
>>>> > 2. To David's point about other transports: in fact currently we are
>>>> using
>>>> > other transports(aside from gRPC) so we don't wanna depend on only
>>>> gRPC
>>>> > features.
>>>> >
>>>> > Cheers,
>>>> > Gosh
>>>> >
>>>> > On Wed, Jun 23, 2021 at 10:40 PM David Li <li...@apache.org>
>>>> wrote:
>>>> >
>>>> > > Thanks for chiming in - I've replied in the doc. Scoping it to just
>>>> > schema
>>>> > > evolution would be preferable, but I'm not sure if Gosh's usecase
>>>> > requires
>>>> > > more flexibility than that or not.
>>>> > >
>>>> > > Again, though, given that 1) gRPC recycles a connection, so repeated
>>>> > calls
>>>> > > aren't necessarily expensive and 2) encoding tricks like
>>>> > union-of-structs,
>>>> > > any solution needs to be weighed against those/we should make sure
>>>> to
>>>> > > document why they aren't sufficient. (For instance, 1) is hampered
>>>> by the
>>>> > > use of L7 load balancers and/or client-side load balancing policies
>>>> in
>>>> > gRPC
>>>> > > and assumes statefulness which is undesirable in general. There's
>>>> also
>>>> > the
>>>> > > eventual desire to have a transport besides gRPC someday.)
>>>> > >
>>>> > > -David
>>>> > >
>>>> > > On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
>>>> > >
>>>> > > Thanks for writing this up! I added a few general comments, but
>>>> have a
>>>> > > question on the approach because it's not quite what I was
>>>> expecting.
>>>> > >
>>>> > > I am slightly concerned that the proposal looks more like support
>>>> for
>>>> > > "multiplexing" IPC streams into a single RPC stream rather than
>>>> support
>>>> > for
>>>> > > a changing Schema of an otherwise consistently logical stream. gRPC
>>>> > already
>>>> > > does a good job decoupling RPC streams from one another. I feel that
>>>> > > throwing that idea into the IPC stream increases client-library
>>>> > > implementation cost by quite a lot.
>>>> > >
>>>> > > Why is it not good enough to replace the Schema when we see a
>>>> duplicate?
>>>> > > This is undoubtedly less work across all client implementations.
>>>> > >
>>>> > > The benefit I see is that you might have two schemas that you swap
>>>> > between
>>>> > > frequently then you can indicate with a single integer. If that's
>>>> what
>>>> > you
>>>> > > want to support I would rather think of them as `schema_id` instead
>>>> of
>>>> > > `stream_id` and not give this impression that multiplexing is a
>>>> goal. As
>>>> > > you have proposed, it seems that the "done writing for a stream"
>>>> needs a
>>>> > > callback notifying the user receiving the stream that a logical
>>>> subset of
>>>> > > the flight is complete. Alternatively, if they aren't independent
>>>> streams
>>>> > > (to the end-user), we could tell the Arrow layer that a particular
>>>> schema
>>>> > > is no longer needed without also needing to communicate further
>>>> > downstream.
>>>> > >
>>>> > > On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org>
>>>> wrote:
>>>> > >
>>>> > > > Ah to be clear, the API is indeed inconsistent - DoExchange was
>>>> added
>>>> > > some
>>>> > > > time later (and by its nature returning a FlightDataStream would
>>>> not
>>>> > have
>>>> > > > been possible, since it's meant to be able to interleave
>>>> > > reading/writing).
>>>> > > > But really, DoGet is indeed the odd one out in the C++ API and it
>>>> may
>>>> > be
>>>> > > > worth correcting. You could also perhaps imagine making a
>>>> > > FlightDataStream
>>>> > > > implementation that accepts a closure and provides it a fake
>>>> writer, if
>>>> > > the
>>>> > > > API mismatch is hard to work with...
>>>> > > >
>>>> > > > That said: this has some benefits, e.g. for a Python service that
>>>> > returns
>>>> > > > a Table, that means data can be fed into gRPC entirely in C++
>>>> without
>>>> > > > having to bounce into Python for each chunk.
>>>> > > >
>>>> > > > Best,
>>>> > > > David
>>>> > > >
>>>> > > > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
>>>> > > > > Hi David,
>>>> > > > >
>>>> > > > > Got you. In fact I was looking at this more from the point of
>>>> view of
>>>> > > > consistency of the API in terms of "inputs" and thought
>>>> DoExchange is
>>>> > > kind
>>>> > > > of a DoGet+ so might make sense to have the same classes being
>>>> utilized
>>>> > > in
>>>> > > > both places. But again, I might be missing something and I get the
>>>> > point
>>>> > > > about breaking change.
>>>> > > > >
>>>> > > > > Cheers,
>>>> > > > > Gosh
>>>> > > > >
>>>> > > > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org>
>>>> > wrote:
>>>> > > > >> __
>>>> > > > >> It's mostly a quirk of implementation (and just for
>>>> clarification,
>>>> > > > they're all nearly identical on the format/protocol level).
>>>> > > > >>
>>>> > > > >> DoGet is conceptualized as your application returning a
>>>> readable
>>>> > > stream
>>>> > > > of batches, instead of your application imperatively writing
>>>> batches to
>>>> > > the
>>>> > > > client. (This is different than how Flight is implemented in
>>>> Java.) You
>>>> > > > would normally not implement FlightDataStream - you would return a
>>>> > > > RecordBatchStream.
>>>> > > > >>
>>>> > > > >> DoGet could not have FlightMessageWriter as a return type as
>>>> that
>>>> > > > wouldn't make sense, but it could accept an instance of that as a
>>>> > > parameter
>>>> > > > instead, much like DoExchange. That would be a breaking change.
>>>> > > > >>
>>>> > > > >> Best,
>>>> > > > >> David
>>>> > > > >>
>>>> > > > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
>>>> > > > >>> Hi David,
>>>> > > > >>>
>>>> > > > >>> Going through the ArrowFlight API: got confused a bit on
>>>> DoGet and
>>>> > > > >>> DoPut/DoExachange apis: the former one expects
>>>> FlightDataStream
>>>> > which
>>>> > > > talks
>>>> > > > >>> in already serialized message terms while the latter to
>>>> > > > >>> accept FlightMessageReader/Writer which expect the user to
>>>> pass in
>>>> > > > >>> RecordBatches etc. Is there any reason why the DoGet can't
>>>> have
>>>> > > > >>> FlightMessageWriter as a return type?
>>>> > > > >>>
>>>> > > > >>> Cheers,
>>>> > > > >>> Gosh
>>>> > > > >>>
>>>> > > > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <
>>>> gosharz@gmail.com
>>>> > >
>>>> > > > wrote:
>>>> > > > >>>
>>>> > > > >>> > Thanks David!
>>>> > > > >>> >
>>>> > > > >>> > I also responded/added more suggestions/questions to the
>>>> doc. I
>>>> > > > think it
>>>> > > > >>> > makes sense to have two sections: one purely protocol
>>>> oriented
>>>> > and
>>>> > > > second
>>>> > > > >>> > API oriented(examples in c++ or in any other language
>>>> should make
>>>> > > > the idea
>>>> > > > >>> > easier to digest).
>>>> > > > >>> >
>>>> > > > >>> > Thanks for the reference too!
>>>> > > > >>> >
>>>> > > > >>> > Cheers,
>>>> > > > >>> > Gosh
>>>> > > > >>> >
>>>> > > > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <
>>>> lidavidm@apache.org>
>>>> > > > wrote:
>>>> > > > >>> >
>>>> > > > >>> >> Thanks! I've left some initial comments/suggestions to
>>>> expand it
>>>> > > in
>>>> > > > terms
>>>> > > > >>> >> of the format definitions and not the C++ APIs.
>>>> > > > >>> >>
>>>> > > > >>> >> I'll also note something like this was proposed a long
>>>> time ago
>>>> > -
>>>> > > > there's
>>>> > > > >>> >> not very much discussion about it there but for reference:
>>>> > > > >>> >>
>>>> > > >
>>>> > >
>>>> >
>>>> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
>>>> > > > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to
>>>> Flight:
>>>> > > > >>> >> "DoBidirectional"' from 2019-2020). It might be good to
>>>> address
>>>> > > why
>>>> > > > the
>>>> > > > >>> >> proposed workaround there (union-of-structs) is
>>>> insufficient for
>>>> > > > the use
>>>> > > > >>> >> cases here (and in FlightSQL).
>>>> > > > >>> >>
>>>> > > > >>> >> -David
>>>> > > > >>> >>
>>>> > > > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
>>>> > > > >>> >> > Ah sorry, comments should work now.
>>>> > > > >>> >> >
>>>> > > > >>> >> > Cheers,
>>>> > > > >>> >> > Gosh
>>>> > > > >>> >> >
>>>> > > > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <
>>>> lidavidm@apache.org
>>>> > > > <mailto:
>>>> > > > >>> >> lidavidm%40apache.org>> wrote:
>>>> > > > >>> >> >
>>>> > > > >>> >> > > Thanks! Will give it a look.
>>>> > > > >>> >> > >
>>>> > > > >>> >> > > Would you mind opening it up for comments?
>>>> > > > >>> >> > >
>>>> > > > >>> >> > > -David
>>>> > > > >>> >> > >
>>>> > > > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <
>>>> gosharz@gmail.com
>>>> > > > <mailto:
>>>> > > > >>> >> gosharz%40gmail.com>> wrote:
>>>> > > > >>> >> > > > Hi folks,
>>>> > > > >>> >> > > >
>>>> > > > >>> >> > > > Started putting some thoughts together here:
>>>> > > > >>> >> > > >
>>>> > > > >>> >> > >
>>>> > > > >>> >>
>>>> > > >
>>>> > >
>>>> >
>>>> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
>>>> > > > >>> >> > > > Any feedback is welcome!
>>>> > > > >>> >> > > >
>>>> > > > >>> >> > > > Cheers,
>>>> > > > >>> >> > > > Gosh
>>>> > > > >>> >> > > >
>>>> > > > >>> >> > >
>>>> > > > >>> >> >
>>>> > > > >>> >>
>>>> > > > >>> >
>>>> > > > >>>
>>>> > > > >>
>>>> > > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > >
>>>> > >
>>>> > >
>>>> >
>>>>
>>>

--

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

Sorry for the second reply:


>    1. In our case we do expect relatively frequent changes in the schema
>    of the batch being sent out. I don't see that pattern changing in the mid
>    term for a good reason. However long term maybe it will be possible to
>    leverage separate RPC calls. I left some description in the comments thread
>    here
>    <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
>    .
>
> I think we should clarify between two types of changes:
1.  Completely new  unique schema needed.
2.  Relatively frequent switches between a few known schemas.

On Fri, Jun 25, 2021 at 2:58 PM Micah Kornfield <em...@gmail.com>
wrote:

>
>>    1. Re complexity of "one schema at a time" vs "schema id based": i
>>    think they are not much different, right? In fact the second one is more of
>>    an optimization to the first one which is beneficial to us. Anyways even
>>    with the first approach you need to add some logic of schema change. Adding
>>    the schema identification is not that complex at first glance.
>>
>> The second approach requires more state being maintained on the client
> side and makes it more difficult to bring schema evolution back into the
> IPC Stream format (i.e. it would live only in flight) without additional
> changes.
>
> In both cases there are also potentially thorny issues related to
> dictionary usages that we will need to resolve.  I haven't thought deeply
> about it but it seems that these might be more complex on a multiplexed
> channel.
>
>
> On Fri, Jun 25, 2021 at 11:52 AM Gosh Arzumanyan <go...@gmail.com>
> wrote:
>
>> Hi Micah,
>>
>> Sure, let me do it here:
>>
>>    1. In our case we do expect relatively frequent changes in the schema
>>    of the batch being sent out. I don't see that pattern changing in the mid
>>    term for a good reason. However long term maybe it will be possible to
>>    leverage separate RPC calls. I left some description in the comments thread
>>    here
>>    <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
>>    .
>>    2. Regarding the union of structs trick: while it is a good
>>    workaround for most of the cases as of now there are also some aspects
>>    which deserve consideration:
>>       1. There is a  space overhead
>>       2. There is an additional glue code to make it work
>>       3. Considering that this solution is bubbling up frequently in
>>       similar contexts, seems like users might benefit if it  was "natively"
>>       supported and they could just focus on populating the schemas they really
>>       need to.
>>    3. Re complexity of "one schema at a time" vs "schema id based": i
>>    think they are not much different, right? In fact the second one is more of
>>    an optimization to the first one which is beneficial to us. Anyways even
>>    with the first approach you need to add some logic of schema change. Adding
>>    the schema identification is not that complex at first glance.
>>
>> Cheers,
>> Gosh
>>
>> On Fri, Jun 25, 2021 at 6:25 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> >
>>> > 1. It seems like renaming stream_id to schema_id and delegating
>>> "logical
>>> > stream" distinction to app_metadata mitigates the "multiplexing" point
>>> > while at the same time it gives enough flexibility to address both
>>> Nate's
>>> > and our use cases.
>>>
>>>
>>> I don't think this is the case.  It seems that having no additional
>>> fields
>>> added and then sending a new schema when necessary combined with Union of
>>> Structs would solve most use-cases.  The main downside could be potential
>>> performance implications if the schema is changing frequently.  Gosh,
>>> could
>>> you address why this wouldn't be sufficient (either here or on the doc).
>>>
>>> Thanks,
>>> -Micah
>>>
>>> On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan <go...@gmail.com>
>>> wrote:
>>>
>>> > Hi guys,
>>> >
>>> > Thanks for sharing your insights/concerns! I also left some comments
>>> based
>>> > on the discussion we had. Briefly:
>>> >
>>> > 1. It seems like renaming stream_id to schema_id and delegating
>>> "logical
>>> > stream" distinction to app_metadata mitigates the "multiplexing" point
>>> > while at the same time it gives enough flexibility to address both
>>> Nate's
>>> > and our use cases.
>>> > 2. To David's point about other transports: in fact currently we are
>>> using
>>> > other transports(aside from gRPC) so we don't wanna depend on only gRPC
>>> > features.
>>> >
>>> > Cheers,
>>> > Gosh
>>> >
>>> > On Wed, Jun 23, 2021 at 10:40 PM David Li <li...@apache.org> wrote:
>>> >
>>> > > Thanks for chiming in - I've replied in the doc. Scoping it to just
>>> > schema
>>> > > evolution would be preferable, but I'm not sure if Gosh's usecase
>>> > requires
>>> > > more flexibility than that or not.
>>> > >
>>> > > Again, though, given that 1) gRPC recycles a connection, so repeated
>>> > calls
>>> > > aren't necessarily expensive and 2) encoding tricks like
>>> > union-of-structs,
>>> > > any solution needs to be weighed against those/we should make sure to
>>> > > document why they aren't sufficient. (For instance, 1) is hampered
>>> by the
>>> > > use of L7 load balancers and/or client-side load balancing policies
>>> in
>>> > gRPC
>>> > > and assumes statefulness which is undesirable in general. There's
>>> also
>>> > the
>>> > > eventual desire to have a transport besides gRPC someday.)
>>> > >
>>> > > -David
>>> > >
>>> > > On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
>>> > >
>>> > > Thanks for writing this up! I added a few general comments, but have
>>> a
>>> > > question on the approach because it's not quite what I was expecting.
>>> > >
>>> > > I am slightly concerned that the proposal looks more like support for
>>> > > "multiplexing" IPC streams into a single RPC stream rather than
>>> support
>>> > for
>>> > > a changing Schema of an otherwise consistently logical stream. gRPC
>>> > already
>>> > > does a good job decoupling RPC streams from one another. I feel that
>>> > > throwing that idea into the IPC stream increases client-library
>>> > > implementation cost by quite a lot.
>>> > >
>>> > > Why is it not good enough to replace the Schema when we see a
>>> duplicate?
>>> > > This is undoubtedly less work across all client implementations.
>>> > >
>>> > > The benefit I see is that you might have two schemas that you swap
>>> > between
>>> > > frequently then you can indicate with a single integer. If that's
>>> what
>>> > you
>>> > > want to support I would rather think of them as `schema_id` instead
>>> of
>>> > > `stream_id` and not give this impression that multiplexing is a
>>> goal. As
>>> > > you have proposed, it seems that the "done writing for a stream"
>>> needs a
>>> > > callback notifying the user receiving the stream that a logical
>>> subset of
>>> > > the flight is complete. Alternatively, if they aren't independent
>>> streams
>>> > > (to the end-user), we could tell the Arrow layer that a particular
>>> schema
>>> > > is no longer needed without also needing to communicate further
>>> > downstream.
>>> > >
>>> > > On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org>
>>> wrote:
>>> > >
>>> > > > Ah to be clear, the API is indeed inconsistent - DoExchange was
>>> added
>>> > > some
>>> > > > time later (and by its nature returning a FlightDataStream would
>>> not
>>> > have
>>> > > > been possible, since it's meant to be able to interleave
>>> > > reading/writing).
>>> > > > But really, DoGet is indeed the odd one out in the C++ API and it
>>> may
>>> > be
>>> > > > worth correcting. You could also perhaps imagine making a
>>> > > FlightDataStream
>>> > > > implementation that accepts a closure and provides it a fake
>>> writer, if
>>> > > the
>>> > > > API mismatch is hard to work with...
>>> > > >
>>> > > > That said: this has some benefits, e.g. for a Python service that
>>> > returns
>>> > > > a Table, that means data can be fed into gRPC entirely in C++
>>> without
>>> > > > having to bounce into Python for each chunk.
>>> > > >
>>> > > > Best,
>>> > > > David
>>> > > >
>>> > > > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
>>> > > > > Hi David,
>>> > > > >
>>> > > > > Got you. In fact I was looking at this more from the point of
>>> view of
>>> > > > consistency of the API in terms of "inputs" and thought DoExchange
>>> is
>>> > > kind
>>> > > > of a DoGet+ so might make sense to have the same classes being
>>> utilized
>>> > > in
>>> > > > both places. But again, I might be missing something and I get the
>>> > point
>>> > > > about breaking change.
>>> > > > >
>>> > > > > Cheers,
>>> > > > > Gosh
>>> > > > >
>>> > > > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org>
>>> > wrote:
>>> > > > >> __
>>> > > > >> It's mostly a quirk of implementation (and just for
>>> clarification,
>>> > > > they're all nearly identical on the format/protocol level).
>>> > > > >>
>>> > > > >> DoGet is conceptualized as your application returning a readable
>>> > > stream
>>> > > > of batches, instead of your application imperatively writing
>>> batches to
>>> > > the
>>> > > > client. (This is different than how Flight is implemented in
>>> Java.) You
>>> > > > would normally not implement FlightDataStream - you would return a
>>> > > > RecordBatchStream.
>>> > > > >>
>>> > > > >> DoGet could not have FlightMessageWriter as a return type as
>>> that
>>> > > > wouldn't make sense, but it could accept an instance of that as a
>>> > > parameter
>>> > > > instead, much like DoExchange. That would be a breaking change.
>>> > > > >>
>>> > > > >> Best,
>>> > > > >> David
>>> > > > >>
>>> > > > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
>>> > > > >>> Hi David,
>>> > > > >>>
>>> > > > >>> Going through the ArrowFlight API: got confused a bit on DoGet
>>> and
>>> > > > >>> DoPut/DoExachange apis: the former one expects FlightDataStream
>>> > which
>>> > > > talks
>>> > > > >>> in already serialized message terms while the latter to
>>> > > > >>> accept FlightMessageReader/Writer which expect the user to
>>> pass in
>>> > > > >>> RecordBatches etc. Is there any reason why the DoGet can't have
>>> > > > >>> FlightMessageWriter as a return type?
>>> > > > >>>
>>> > > > >>> Cheers,
>>> > > > >>> Gosh
>>> > > > >>>
>>> > > > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <
>>> gosharz@gmail.com
>>> > >
>>> > > > wrote:
>>> > > > >>>
>>> > > > >>> > Thanks David!
>>> > > > >>> >
>>> > > > >>> > I also responded/added more suggestions/questions to the
>>> doc. I
>>> > > > think it
>>> > > > >>> > makes sense to have two sections: one purely protocol
>>> oriented
>>> > and
>>> > > > second
>>> > > > >>> > API oriented(examples in c++ or in any other language should
>>> make
>>> > > > the idea
>>> > > > >>> > easier to digest).
>>> > > > >>> >
>>> > > > >>> > Thanks for the reference too!
>>> > > > >>> >
>>> > > > >>> > Cheers,
>>> > > > >>> > Gosh
>>> > > > >>> >
>>> > > > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <
>>> lidavidm@apache.org>
>>> > > > wrote:
>>> > > > >>> >
>>> > > > >>> >> Thanks! I've left some initial comments/suggestions to
>>> expand it
>>> > > in
>>> > > > terms
>>> > > > >>> >> of the format definitions and not the C++ APIs.
>>> > > > >>> >>
>>> > > > >>> >> I'll also note something like this was proposed a long time
>>> ago
>>> > -
>>> > > > there's
>>> > > > >>> >> not very much discussion about it there but for reference:
>>> > > > >>> >>
>>> > > >
>>> > >
>>> >
>>> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
>>> > > > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to
>>> Flight:
>>> > > > >>> >> "DoBidirectional"' from 2019-2020). It might be good to
>>> address
>>> > > why
>>> > > > the
>>> > > > >>> >> proposed workaround there (union-of-structs) is
>>> insufficient for
>>> > > > the use
>>> > > > >>> >> cases here (and in FlightSQL).
>>> > > > >>> >>
>>> > > > >>> >> -David
>>> > > > >>> >>
>>> > > > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
>>> > > > >>> >> > Ah sorry, comments should work now.
>>> > > > >>> >> >
>>> > > > >>> >> > Cheers,
>>> > > > >>> >> > Gosh
>>> > > > >>> >> >
>>> > > > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <
>>> lidavidm@apache.org
>>> > > > <mailto:
>>> > > > >>> >> lidavidm%40apache.org>> wrote:
>>> > > > >>> >> >
>>> > > > >>> >> > > Thanks! Will give it a look.
>>> > > > >>> >> > >
>>> > > > >>> >> > > Would you mind opening it up for comments?
>>> > > > >>> >> > >
>>> > > > >>> >> > > -David
>>> > > > >>> >> > >
>>> > > > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <
>>> gosharz@gmail.com
>>> > > > <mailto:
>>> > > > >>> >> gosharz%40gmail.com>> wrote:
>>> > > > >>> >> > > > Hi folks,
>>> > > > >>> >> > > >
>>> > > > >>> >> > > > Started putting some thoughts together here:
>>> > > > >>> >> > > >
>>> > > > >>> >> > >
>>> > > > >>> >>
>>> > > >
>>> > >
>>> >
>>> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
>>> > > > >>> >> > > > Any feedback is welcome!
>>> > > > >>> >> > > >
>>> > > > >>> >> > > > Cheers,
>>> > > > >>> >> > > > Gosh
>>> > > > >>> >> > > >
>>> > > > >>> >> > >
>>> > > > >>> >> >
>>> > > > >>> >>
>>> > > > >>> >
>>> > > > >>>
>>> > > > >>
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

>
>
>    1. Re complexity of "one schema at a time" vs "schema id based": i
>    think they are not much different, right? In fact the second one is more of
>    an optimization to the first one which is beneficial to us. Anyways even
>    with the first approach you need to add some logic of schema change. Adding
>    the schema identification is not that complex at first glance.
>
> The second approach requires more state being maintained on the client
side and makes it more difficult to bring schema evolution back into the
IPC Stream format (i.e. it would live only in flight) without additional
changes.

In both cases there are also potentially thorny issues related to
dictionary usages that we will need to resolve.  I haven't thought deeply
about it but it seems that these might be more complex on a multiplexed
channel.


On Fri, Jun 25, 2021 at 11:52 AM Gosh Arzumanyan <go...@gmail.com> wrote:

> Hi Micah,
>
> Sure, let me do it here:
>
>    1. In our case we do expect relatively frequent changes in the schema
>    of the batch being sent out. I don't see that pattern changing in the mid
>    term for a good reason. However long term maybe it will be possible to
>    leverage separate RPC calls. I left some description in the comments thread
>    here
>    <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
>    .
>    2. Regarding the union of structs trick: while it is a good workaround
>    for most of the cases as of now there are also some aspects which deserve
>    consideration:
>       1. There is a  space overhead
>       2. There is an additional glue code to make it work
>       3. Considering that this solution is bubbling up frequently in
>       similar contexts, seems like users might benefit if it  was "natively"
>       supported and they could just focus on populating the schemas they really
>       need to.
>    3. Re complexity of "one schema at a time" vs "schema id based": i
>    think they are not much different, right? In fact the second one is more of
>    an optimization to the first one which is beneficial to us. Anyways even
>    with the first approach you need to add some logic of schema change. Adding
>    the schema identification is not that complex at first glance.
>
> Cheers,
> Gosh
>
> On Fri, Jun 25, 2021 at 6:25 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> >
>> > 1. It seems like renaming stream_id to schema_id and delegating "logical
>> > stream" distinction to app_metadata mitigates the "multiplexing" point
>> > while at the same time it gives enough flexibility to address both
>> Nate's
>> > and our use cases.
>>
>>
>> I don't think this is the case.  It seems that having no additional fields
>> added and then sending a new schema when necessary combined with Union of
>> Structs would solve most use-cases.  The main downside could be potential
>> performance implications if the schema is changing frequently.  Gosh,
>> could
>> you address why this wouldn't be sufficient (either here or on the doc).
>>
>> Thanks,
>> -Micah
>>
>> On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan <go...@gmail.com>
>> wrote:
>>
>> > Hi guys,
>> >
>> > Thanks for sharing your insights/concerns! I also left some comments
>> based
>> > on the discussion we had. Briefly:
>> >
>> > 1. It seems like renaming stream_id to schema_id and delegating "logical
>> > stream" distinction to app_metadata mitigates the "multiplexing" point
>> > while at the same time it gives enough flexibility to address both
>> Nate's
>> > and our use cases.
>> > 2. To David's point about other transports: in fact currently we are
>> using
>> > other transports(aside from gRPC) so we don't wanna depend on only gRPC
>> > features.
>> >
>> > Cheers,
>> > Gosh
>> >
>> > On Wed, Jun 23, 2021 at 10:40 PM David Li <li...@apache.org> wrote:
>> >
>> > > Thanks for chiming in - I've replied in the doc. Scoping it to just
>> > schema
>> > > evolution would be preferable, but I'm not sure if Gosh's usecase
>> > requires
>> > > more flexibility than that or not.
>> > >
>> > > Again, though, given that 1) gRPC recycles a connection, so repeated
>> > calls
>> > > aren't necessarily expensive and 2) encoding tricks like
>> > union-of-structs,
>> > > any solution needs to be weighed against those/we should make sure to
>> > > document why they aren't sufficient. (For instance, 1) is hampered by
>> the
>> > > use of L7 load balancers and/or client-side load balancing policies in
>> > gRPC
>> > > and assumes statefulness which is undesirable in general. There's also
>> > the
>> > > eventual desire to have a transport besides gRPC someday.)
>> > >
>> > > -David
>> > >
>> > > On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
>> > >
>> > > Thanks for writing this up! I added a few general comments, but have a
>> > > question on the approach because it's not quite what I was expecting.
>> > >
>> > > I am slightly concerned that the proposal looks more like support for
>> > > "multiplexing" IPC streams into a single RPC stream rather than
>> support
>> > for
>> > > a changing Schema of an otherwise consistently logical stream. gRPC
>> > already
>> > > does a good job decoupling RPC streams from one another. I feel that
>> > > throwing that idea into the IPC stream increases client-library
>> > > implementation cost by quite a lot.
>> > >
>> > > Why is it not good enough to replace the Schema when we see a
>> duplicate?
>> > > This is undoubtedly less work across all client implementations.
>> > >
>> > > The benefit I see is that you might have two schemas that you swap
>> > between
>> > > frequently then you can indicate with a single integer. If that's what
>> > you
>> > > want to support I would rather think of them as `schema_id` instead of
>> > > `stream_id` and not give this impression that multiplexing is a goal.
>> As
>> > > you have proposed, it seems that the "done writing for a stream"
>> needs a
>> > > callback notifying the user receiving the stream that a logical
>> subset of
>> > > the flight is complete. Alternatively, if they aren't independent
>> streams
>> > > (to the end-user), we could tell the Arrow layer that a particular
>> schema
>> > > is no longer needed without also needing to communicate further
>> > downstream.
>> > >
>> > > On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org> wrote:
>> > >
>> > > > Ah to be clear, the API is indeed inconsistent - DoExchange was
>> added
>> > > some
>> > > > time later (and by its nature returning a FlightDataStream would not
>> > have
>> > > > been possible, since it's meant to be able to interleave
>> > > reading/writing).
>> > > > But really, DoGet is indeed the odd one out in the C++ API and it
>> may
>> > be
>> > > > worth correcting. You could also perhaps imagine making a
>> > > FlightDataStream
>> > > > implementation that accepts a closure and provides it a fake
>> writer, if
>> > > the
>> > > > API mismatch is hard to work with...
>> > > >
>> > > > That said: this has some benefits, e.g. for a Python service that
>> > returns
>> > > > a Table, that means data can be fed into gRPC entirely in C++
>> without
>> > > > having to bounce into Python for each chunk.
>> > > >
>> > > > Best,
>> > > > David
>> > > >
>> > > > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
>> > > > > Hi David,
>> > > > >
>> > > > > Got you. In fact I was looking at this more from the point of
>> view of
>> > > > consistency of the API in terms of "inputs" and thought DoExchange
>> is
>> > > kind
>> > > > of a DoGet+ so might make sense to have the same classes being
>> utilized
>> > > in
>> > > > both places. But again, I might be missing something and I get the
>> > point
>> > > > about breaking change.
>> > > > >
>> > > > > Cheers,
>> > > > > Gosh
>> > > > >
>> > > > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org>
>> > wrote:
>> > > > >> __
>> > > > >> It's mostly a quirk of implementation (and just for
>> clarification,
>> > > > they're all nearly identical on the format/protocol level).
>> > > > >>
>> > > > >> DoGet is conceptualized as your application returning a readable
>> > > stream
>> > > > of batches, instead of your application imperatively writing
>> batches to
>> > > the
>> > > > client. (This is different than how Flight is implemented in Java.)
>> You
>> > > > would normally not implement FlightDataStream - you would return a
>> > > > RecordBatchStream.
>> > > > >>
>> > > > >> DoGet could not have FlightMessageWriter as a return type as that
>> > > > wouldn't make sense, but it could accept an instance of that as a
>> > > parameter
>> > > > instead, much like DoExchange. That would be a breaking change.
>> > > > >>
>> > > > >> Best,
>> > > > >> David
>> > > > >>
>> > > > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
>> > > > >>> Hi David,
>> > > > >>>
>> > > > >>> Going through the ArrowFlight API: got confused a bit on DoGet
>> and
>> > > > >>> DoPut/DoExachange apis: the former one expects FlightDataStream
>> > which
>> > > > talks
>> > > > >>> in already serialized message terms while the latter to
>> > > > >>> accept FlightMessageReader/Writer which expect the user to pass
>> in
>> > > > >>> RecordBatches etc. Is there any reason why the DoGet can't have
>> > > > >>> FlightMessageWriter as a return type?
>> > > > >>>
>> > > > >>> Cheers,
>> > > > >>> Gosh
>> > > > >>>
>> > > > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <
>> gosharz@gmail.com
>> > >
>> > > > wrote:
>> > > > >>>
>> > > > >>> > Thanks David!
>> > > > >>> >
>> > > > >>> > I also responded/added more suggestions/questions to the doc.
>> I
>> > > > think it
>> > > > >>> > makes sense to have two sections: one purely protocol oriented
>> > and
>> > > > second
>> > > > >>> > API oriented(examples in c++ or in any other language should
>> make
>> > > > the idea
>> > > > >>> > easier to digest).
>> > > > >>> >
>> > > > >>> > Thanks for the reference too!
>> > > > >>> >
>> > > > >>> > Cheers,
>> > > > >>> > Gosh
>> > > > >>> >
>> > > > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <lidavidm@apache.org
>> >
>> > > > wrote:
>> > > > >>> >
>> > > > >>> >> Thanks! I've left some initial comments/suggestions to
>> expand it
>> > > in
>> > > > terms
>> > > > >>> >> of the format definitions and not the C++ APIs.
>> > > > >>> >>
>> > > > >>> >> I'll also note something like this was proposed a long time
>> ago
>> > -
>> > > > there's
>> > > > >>> >> not very much discussion about it there but for reference:
>> > > > >>> >>
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
>> > > > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to
>> Flight:
>> > > > >>> >> "DoBidirectional"' from 2019-2020). It might be good to
>> address
>> > > why
>> > > > the
>> > > > >>> >> proposed workaround there (union-of-structs) is insufficient
>> for
>> > > > the use
>> > > > >>> >> cases here (and in FlightSQL).
>> > > > >>> >>
>> > > > >>> >> -David
>> > > > >>> >>
>> > > > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
>> > > > >>> >> > Ah sorry, comments should work now.
>> > > > >>> >> >
>> > > > >>> >> > Cheers,
>> > > > >>> >> > Gosh
>> > > > >>> >> >
>> > > > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <
>> lidavidm@apache.org
>> > > > <mailto:
>> > > > >>> >> lidavidm%40apache.org>> wrote:
>> > > > >>> >> >
>> > > > >>> >> > > Thanks! Will give it a look.
>> > > > >>> >> > >
>> > > > >>> >> > > Would you mind opening it up for comments?
>> > > > >>> >> > >
>> > > > >>> >> > > -David
>> > > > >>> >> > >
>> > > > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <
>> gosharz@gmail.com
>> > > > <mailto:
>> > > > >>> >> gosharz%40gmail.com>> wrote:
>> > > > >>> >> > > > Hi folks,
>> > > > >>> >> > > >
>> > > > >>> >> > > > Started putting some thoughts together here:
>> > > > >>> >> > > >
>> > > > >>> >> > >
>> > > > >>> >>
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
>> > > > >>> >> > > > Any feedback is welcome!
>> > > > >>> >> > > >
>> > > > >>> >> > > > Cheers,
>> > > > >>> >> > > > Gosh
>> > > > >>> >> > > >
>> > > > >>> >> > >
>> > > > >>> >> >
>> > > > >>> >>
>> > > > >>> >
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> > >
>> > > --
>> > >
>> > >
>> > >
>> >
>>
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi Micah,

Sure, let me do it here:

   1. In our case we do expect relatively frequent changes in the schema of
   the batch being sent out. I don't see that pattern changing in the mid term
   for a good reason. However long term maybe it will be possible to leverage
   separate RPC calls. I left some description in the comments thread here
   <https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?disco=AAAAMkkJ9xQ>
   .
   2. Regarding the union of structs trick: while it is a good workaround
   for most of the cases as of now there are also some aspects which deserve
   consideration:
      1. There is a  space overhead
      2. There is an additional glue code to make it work
      3. Considering that this solution is bubbling up frequently in
      similar contexts, seems like users might benefit if it  was "natively"
      supported and they could just focus on populating the schemas they really
      need to.
   3. Re complexity of "one schema at a time" vs "schema id based": i think
   they are not much different, right? In fact the second one is more of an
   optimization to the first one which is beneficial to us. Anyways even with
   the first approach you need to add some logic of schema change. Adding the
   schema identification is not that complex at first glance.

Cheers,
Gosh

On Fri, Jun 25, 2021 at 6:25 PM Micah Kornfield <em...@gmail.com>
wrote:

> >
> > 1. It seems like renaming stream_id to schema_id and delegating "logical
> > stream" distinction to app_metadata mitigates the "multiplexing" point
> > while at the same time it gives enough flexibility to address both Nate's
> > and our use cases.
>
>
> I don't think this is the case.  It seems that having no additional fields
> added and then sending a new schema when necessary combined with Union of
> Structs would solve most use-cases.  The main downside could be potential
> performance implications if the schema is changing frequently.  Gosh, could
> you address why this wouldn't be sufficient (either here or on the doc).
>
> Thanks,
> -Micah
>
> On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan <go...@gmail.com> wrote:
>
> > Hi guys,
> >
> > Thanks for sharing your insights/concerns! I also left some comments
> based
> > on the discussion we had. Briefly:
> >
> > 1. It seems like renaming stream_id to schema_id and delegating "logical
> > stream" distinction to app_metadata mitigates the "multiplexing" point
> > while at the same time it gives enough flexibility to address both Nate's
> > and our use cases.
> > 2. To David's point about other transports: in fact currently we are
> using
> > other transports(aside from gRPC) so we don't wanna depend on only gRPC
> > features.
> >
> > Cheers,
> > Gosh
> >
> > On Wed, Jun 23, 2021 at 10:40 PM David Li <li...@apache.org> wrote:
> >
> > > Thanks for chiming in - I've replied in the doc. Scoping it to just
> > schema
> > > evolution would be preferable, but I'm not sure if Gosh's usecase
> > requires
> > > more flexibility than that or not.
> > >
> > > Again, though, given that 1) gRPC recycles a connection, so repeated
> > calls
> > > aren't necessarily expensive and 2) encoding tricks like
> > union-of-structs,
> > > any solution needs to be weighed against those/we should make sure to
> > > document why they aren't sufficient. (For instance, 1) is hampered by
> the
> > > use of L7 load balancers and/or client-side load balancing policies in
> > gRPC
> > > and assumes statefulness which is undesirable in general. There's also
> > the
> > > eventual desire to have a transport besides gRPC someday.)
> > >
> > > -David
> > >
> > > On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
> > >
> > > Thanks for writing this up! I added a few general comments, but have a
> > > question on the approach because it's not quite what I was expecting.
> > >
> > > I am slightly concerned that the proposal looks more like support for
> > > "multiplexing" IPC streams into a single RPC stream rather than support
> > for
> > > a changing Schema of an otherwise consistently logical stream. gRPC
> > already
> > > does a good job decoupling RPC streams from one another. I feel that
> > > throwing that idea into the IPC stream increases client-library
> > > implementation cost by quite a lot.
> > >
> > > Why is it not good enough to replace the Schema when we see a
> duplicate?
> > > This is undoubtedly less work across all client implementations.
> > >
> > > The benefit I see is that you might have two schemas that you swap
> > between
> > > frequently then you can indicate with a single integer. If that's what
> > you
> > > want to support I would rather think of them as `schema_id` instead of
> > > `stream_id` and not give this impression that multiplexing is a goal.
> As
> > > you have proposed, it seems that the "done writing for a stream" needs
> a
> > > callback notifying the user receiving the stream that a logical subset
> of
> > > the flight is complete. Alternatively, if they aren't independent
> streams
> > > (to the end-user), we could tell the Arrow layer that a particular
> schema
> > > is no longer needed without also needing to communicate further
> > downstream.
> > >
> > > On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org> wrote:
> > >
> > > > Ah to be clear, the API is indeed inconsistent - DoExchange was added
> > > some
> > > > time later (and by its nature returning a FlightDataStream would not
> > have
> > > > been possible, since it's meant to be able to interleave
> > > reading/writing).
> > > > But really, DoGet is indeed the odd one out in the C++ API and it may
> > be
> > > > worth correcting. You could also perhaps imagine making a
> > > FlightDataStream
> > > > implementation that accepts a closure and provides it a fake writer,
> if
> > > the
> > > > API mismatch is hard to work with...
> > > >
> > > > That said: this has some benefits, e.g. for a Python service that
> > returns
> > > > a Table, that means data can be fed into gRPC entirely in C++ without
> > > > having to bounce into Python for each chunk.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
> > > > > Hi David,
> > > > >
> > > > > Got you. In fact I was looking at this more from the point of view
> of
> > > > consistency of the API in terms of "inputs" and thought DoExchange is
> > > kind
> > > > of a DoGet+ so might make sense to have the same classes being
> utilized
> > > in
> > > > both places. But again, I might be missing something and I get the
> > point
> > > > about breaking change.
> > > > >
> > > > > Cheers,
> > > > > Gosh
> > > > >
> > > > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org>
> > wrote:
> > > > >> __
> > > > >> It's mostly a quirk of implementation (and just for clarification,
> > > > they're all nearly identical on the format/protocol level).
> > > > >>
> > > > >> DoGet is conceptualized as your application returning a readable
> > > stream
> > > > of batches, instead of your application imperatively writing batches
> to
> > > the
> > > > client. (This is different than how Flight is implemented in Java.)
> You
> > > > would normally not implement FlightDataStream - you would return a
> > > > RecordBatchStream.
> > > > >>
> > > > >> DoGet could not have FlightMessageWriter as a return type as that
> > > > wouldn't make sense, but it could accept an instance of that as a
> > > parameter
> > > > instead, much like DoExchange. That would be a breaking change.
> > > > >>
> > > > >> Best,
> > > > >> David
> > > > >>
> > > > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
> > > > >>> Hi David,
> > > > >>>
> > > > >>> Going through the ArrowFlight API: got confused a bit on DoGet
> and
> > > > >>> DoPut/DoExachange apis: the former one expects FlightDataStream
> > which
> > > > talks
> > > > >>> in already serialized message terms while the latter to
> > > > >>> accept FlightMessageReader/Writer which expect the user to pass
> in
> > > > >>> RecordBatches etc. Is there any reason why the DoGet can't have
> > > > >>> FlightMessageWriter as a return type?
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Gosh
> > > > >>>
> > > > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <
> gosharz@gmail.com
> > >
> > > > wrote:
> > > > >>>
> > > > >>> > Thanks David!
> > > > >>> >
> > > > >>> > I also responded/added more suggestions/questions to the doc. I
> > > > think it
> > > > >>> > makes sense to have two sections: one purely protocol oriented
> > and
> > > > second
> > > > >>> > API oriented(examples in c++ or in any other language should
> make
> > > > the idea
> > > > >>> > easier to digest).
> > > > >>> >
> > > > >>> > Thanks for the reference too!
> > > > >>> >
> > > > >>> > Cheers,
> > > > >>> > Gosh
> > > > >>> >
> > > > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org>
> > > > wrote:
> > > > >>> >
> > > > >>> >> Thanks! I've left some initial comments/suggestions to expand
> it
> > > in
> > > > terms
> > > > >>> >> of the format definitions and not the C++ APIs.
> > > > >>> >>
> > > > >>> >> I'll also note something like this was proposed a long time
> ago
> > -
> > > > there's
> > > > >>> >> not very much discussion about it there but for reference:
> > > > >>> >>
> > > >
> > >
> >
> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> > > > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> > > > >>> >> "DoBidirectional"' from 2019-2020). It might be good to
> address
> > > why
> > > > the
> > > > >>> >> proposed workaround there (union-of-structs) is insufficient
> for
> > > > the use
> > > > >>> >> cases here (and in FlightSQL).
> > > > >>> >>
> > > > >>> >> -David
> > > > >>> >>
> > > > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> > > > >>> >> > Ah sorry, comments should work now.
> > > > >>> >> >
> > > > >>> >> > Cheers,
> > > > >>> >> > Gosh
> > > > >>> >> >
> > > > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org
> > > > <mailto:
> > > > >>> >> lidavidm%40apache.org>> wrote:
> > > > >>> >> >
> > > > >>> >> > > Thanks! Will give it a look.
> > > > >>> >> > >
> > > > >>> >> > > Would you mind opening it up for comments?
> > > > >>> >> > >
> > > > >>> >> > > -David
> > > > >>> >> > >
> > > > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <
> gosharz@gmail.com
> > > > <mailto:
> > > > >>> >> gosharz%40gmail.com>> wrote:
> > > > >>> >> > > > Hi folks,
> > > > >>> >> > > >
> > > > >>> >> > > > Started putting some thoughts together here:
> > > > >>> >> > > >
> > > > >>> >> > >
> > > > >>> >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> > > > >>> >> > > > Any feedback is welcome!
> > > > >>> >> > > >
> > > > >>> >> > > > Cheers,
> > > > >>> >> > > > Gosh
> > > > >>> >> > > >
> > > > >>> >> > >
> > > > >>> >> >
> > > > >>> >>
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> > >
> > > --
> > >
> > >
> > >
> >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Micah Kornfield <em...@gmail.com>]

>
> 1. It seems like renaming stream_id to schema_id and delegating "logical
> stream" distinction to app_metadata mitigates the "multiplexing" point
> while at the same time it gives enough flexibility to address both Nate's
> and our use cases.


I don't think this is the case.  It seems that having no additional fields
added and then sending a new schema when necessary combined with Union of
Structs would solve most use-cases.  The main downside could be potential
performance implications if the schema is changing frequently.  Gosh, could
you address why this wouldn't be sufficient (either here or on the doc).

Thanks,
-Micah

On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan <go...@gmail.com> wrote:

> Hi guys,
>
> Thanks for sharing your insights/concerns! I also left some comments based
> on the discussion we had. Briefly:
>
> 1. It seems like renaming stream_id to schema_id and delegating "logical
> stream" distinction to app_metadata mitigates the "multiplexing" point
> while at the same time it gives enough flexibility to address both Nate's
> and our use cases.
> 2. To David's point about other transports: in fact currently we are using
> other transports(aside from gRPC) so we don't wanna depend on only gRPC
> features.
>
> Cheers,
> Gosh
>
> On Wed, Jun 23, 2021 at 10:40 PM David Li <li...@apache.org> wrote:
>
> > Thanks for chiming in - I've replied in the doc. Scoping it to just
> schema
> > evolution would be preferable, but I'm not sure if Gosh's usecase
> requires
> > more flexibility than that or not.
> >
> > Again, though, given that 1) gRPC recycles a connection, so repeated
> calls
> > aren't necessarily expensive and 2) encoding tricks like
> union-of-structs,
> > any solution needs to be weighed against those/we should make sure to
> > document why they aren't sufficient. (For instance, 1) is hampered by the
> > use of L7 load balancers and/or client-side load balancing policies in
> gRPC
> > and assumes statefulness which is undesirable in general. There's also
> the
> > eventual desire to have a transport besides gRPC someday.)
> >
> > -David
> >
> > On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
> >
> > Thanks for writing this up! I added a few general comments, but have a
> > question on the approach because it's not quite what I was expecting.
> >
> > I am slightly concerned that the proposal looks more like support for
> > "multiplexing" IPC streams into a single RPC stream rather than support
> for
> > a changing Schema of an otherwise consistently logical stream. gRPC
> already
> > does a good job decoupling RPC streams from one another. I feel that
> > throwing that idea into the IPC stream increases client-library
> > implementation cost by quite a lot.
> >
> > Why is it not good enough to replace the Schema when we see a duplicate?
> > This is undoubtedly less work across all client implementations.
> >
> > The benefit I see is that you might have two schemas that you swap
> between
> > frequently then you can indicate with a single integer. If that's what
> you
> > want to support I would rather think of them as `schema_id` instead of
> > `stream_id` and not give this impression that multiplexing is a goal. As
> > you have proposed, it seems that the "done writing for a stream" needs a
> > callback notifying the user receiving the stream that a logical subset of
> > the flight is complete. Alternatively, if they aren't independent streams
> > (to the end-user), we could tell the Arrow layer that a particular schema
> > is no longer needed without also needing to communicate further
> downstream.
> >
> > On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org> wrote:
> >
> > > Ah to be clear, the API is indeed inconsistent - DoExchange was added
> > some
> > > time later (and by its nature returning a FlightDataStream would not
> have
> > > been possible, since it's meant to be able to interleave
> > reading/writing).
> > > But really, DoGet is indeed the odd one out in the C++ API and it may
> be
> > > worth correcting. You could also perhaps imagine making a
> > FlightDataStream
> > > implementation that accepts a closure and provides it a fake writer, if
> > the
> > > API mismatch is hard to work with...
> > >
> > > That said: this has some benefits, e.g. for a Python service that
> returns
> > > a Table, that means data can be fed into gRPC entirely in C++ without
> > > having to bounce into Python for each chunk.
> > >
> > > Best,
> > > David
> > >
> > > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
> > > > Hi David,
> > > >
> > > > Got you. In fact I was looking at this more from the point of view of
> > > consistency of the API in terms of "inputs" and thought DoExchange is
> > kind
> > > of a DoGet+ so might make sense to have the same classes being utilized
> > in
> > > both places. But again, I might be missing something and I get the
> point
> > > about breaking change.
> > > >
> > > > Cheers,
> > > > Gosh
> > > >
> > > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org>
> wrote:
> > > >> __
> > > >> It's mostly a quirk of implementation (and just for clarification,
> > > they're all nearly identical on the format/protocol level).
> > > >>
> > > >> DoGet is conceptualized as your application returning a readable
> > stream
> > > of batches, instead of your application imperatively writing batches to
> > the
> > > client. (This is different than how Flight is implemented in Java.) You
> > > would normally not implement FlightDataStream - you would return a
> > > RecordBatchStream.
> > > >>
> > > >> DoGet could not have FlightMessageWriter as a return type as that
> > > wouldn't make sense, but it could accept an instance of that as a
> > parameter
> > > instead, much like DoExchange. That would be a breaking change.
> > > >>
> > > >> Best,
> > > >> David
> > > >>
> > > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
> > > >>> Hi David,
> > > >>>
> > > >>> Going through the ArrowFlight API: got confused a bit on DoGet and
> > > >>> DoPut/DoExachange apis: the former one expects FlightDataStream
> which
> > > talks
> > > >>> in already serialized message terms while the latter to
> > > >>> accept FlightMessageReader/Writer which expect the user to pass in
> > > >>> RecordBatches etc. Is there any reason why the DoGet can't have
> > > >>> FlightMessageWriter as a return type?
> > > >>>
> > > >>> Cheers,
> > > >>> Gosh
> > > >>>
> > > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <gosharz@gmail.com
> >
> > > wrote:
> > > >>>
> > > >>> > Thanks David!
> > > >>> >
> > > >>> > I also responded/added more suggestions/questions to the doc. I
> > > think it
> > > >>> > makes sense to have two sections: one purely protocol oriented
> and
> > > second
> > > >>> > API oriented(examples in c++ or in any other language should make
> > > the idea
> > > >>> > easier to digest).
> > > >>> >
> > > >>> > Thanks for the reference too!
> > > >>> >
> > > >>> > Cheers,
> > > >>> > Gosh
> > > >>> >
> > > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org>
> > > wrote:
> > > >>> >
> > > >>> >> Thanks! I've left some initial comments/suggestions to expand it
> > in
> > > terms
> > > >>> >> of the format definitions and not the C++ APIs.
> > > >>> >>
> > > >>> >> I'll also note something like this was proposed a long time ago
> -
> > > there's
> > > >>> >> not very much discussion about it there but for reference:
> > > >>> >>
> > >
> >
> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> > > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> > > >>> >> "DoBidirectional"' from 2019-2020). It might be good to address
> > why
> > > the
> > > >>> >> proposed workaround there (union-of-structs) is insufficient for
> > > the use
> > > >>> >> cases here (and in FlightSQL).
> > > >>> >>
> > > >>> >> -David
> > > >>> >>
> > > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> > > >>> >> > Ah sorry, comments should work now.
> > > >>> >> >
> > > >>> >> > Cheers,
> > > >>> >> > Gosh
> > > >>> >> >
> > > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org
> > > <mailto:
> > > >>> >> lidavidm%40apache.org>> wrote:
> > > >>> >> >
> > > >>> >> > > Thanks! Will give it a look.
> > > >>> >> > >
> > > >>> >> > > Would you mind opening it up for comments?
> > > >>> >> > >
> > > >>> >> > > -David
> > > >>> >> > >
> > > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com
> > > <mailto:
> > > >>> >> gosharz%40gmail.com>> wrote:
> > > >>> >> > > > Hi folks,
> > > >>> >> > > >
> > > >>> >> > > > Started putting some thoughts together here:
> > > >>> >> > > >
> > > >>> >> > >
> > > >>> >>
> > >
> >
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> > > >>> >> > > > Any feedback is welcome!
> > > >>> >> > > >
> > > >>> >> > > > Cheers,
> > > >>> >> > > > Gosh
> > > >>> >> > > >
> > > >>> >> > >
> > > >>> >> >
> > > >>> >>
> > > >>> >
> > > >>>
> > > >>
> > >
> >
> >
> > --
> >
> >
> >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi guys,

Thanks for sharing your insights/concerns! I also left some comments based
on the discussion we had. Briefly:

1. It seems like renaming stream_id to schema_id and delegating "logical
stream" distinction to app_metadata mitigates the "multiplexing" point
while at the same time it gives enough flexibility to address both Nate's
and our use cases.
2. To David's point about other transports: in fact currently we are using
other transports(aside from gRPC) so we don't wanna depend on only gRPC
features.

Cheers,
Gosh

On Wed, Jun 23, 2021 at 10:40 PM David Li <li...@apache.org> wrote:

> Thanks for chiming in - I've replied in the doc. Scoping it to just schema
> evolution would be preferable, but I'm not sure if Gosh's usecase requires
> more flexibility than that or not.
>
> Again, though, given that 1) gRPC recycles a connection, so repeated calls
> aren't necessarily expensive and 2) encoding tricks like union-of-structs,
> any solution needs to be weighed against those/we should make sure to
> document why they aren't sufficient. (For instance, 1) is hampered by the
> use of L7 load balancers and/or client-side load balancing policies in gRPC
> and assumes statefulness which is undesirable in general. There's also the
> eventual desire to have a transport besides gRPC someday.)
>
> -David
>
> On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
>
> Thanks for writing this up! I added a few general comments, but have a
> question on the approach because it's not quite what I was expecting.
>
> I am slightly concerned that the proposal looks more like support for
> "multiplexing" IPC streams into a single RPC stream rather than support for
> a changing Schema of an otherwise consistently logical stream. gRPC already
> does a good job decoupling RPC streams from one another. I feel that
> throwing that idea into the IPC stream increases client-library
> implementation cost by quite a lot.
>
> Why is it not good enough to replace the Schema when we see a duplicate?
> This is undoubtedly less work across all client implementations.
>
> The benefit I see is that you might have two schemas that you swap between
> frequently then you can indicate with a single integer. If that's what you
> want to support I would rather think of them as `schema_id` instead of
> `stream_id` and not give this impression that multiplexing is a goal. As
> you have proposed, it seems that the "done writing for a stream" needs a
> callback notifying the user receiving the stream that a logical subset of
> the flight is complete. Alternatively, if they aren't independent streams
> (to the end-user), we could tell the Arrow layer that a particular schema
> is no longer needed without also needing to communicate further downstream.
>
> On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org> wrote:
>
> > Ah to be clear, the API is indeed inconsistent - DoExchange was added
> some
> > time later (and by its nature returning a FlightDataStream would not have
> > been possible, since it's meant to be able to interleave
> reading/writing).
> > But really, DoGet is indeed the odd one out in the C++ API and it may be
> > worth correcting. You could also perhaps imagine making a
> FlightDataStream
> > implementation that accepts a closure and provides it a fake writer, if
> the
> > API mismatch is hard to work with...
> >
> > That said: this has some benefits, e.g. for a Python service that returns
> > a Table, that means data can be fed into gRPC entirely in C++ without
> > having to bounce into Python for each chunk.
> >
> > Best,
> > David
> >
> > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
> > > Hi David,
> > >
> > > Got you. In fact I was looking at this more from the point of view of
> > consistency of the API in terms of "inputs" and thought DoExchange is
> kind
> > of a DoGet+ so might make sense to have the same classes being utilized
> in
> > both places. But again, I might be missing something and I get the point
> > about breaking change.
> > >
> > > Cheers,
> > > Gosh
> > >
> > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org> wrote:
> > >> __
> > >> It's mostly a quirk of implementation (and just for clarification,
> > they're all nearly identical on the format/protocol level).
> > >>
> > >> DoGet is conceptualized as your application returning a readable
> stream
> > of batches, instead of your application imperatively writing batches to
> the
> > client. (This is different than how Flight is implemented in Java.) You
> > would normally not implement FlightDataStream - you would return a
> > RecordBatchStream.
> > >>
> > >> DoGet could not have FlightMessageWriter as a return type as that
> > wouldn't make sense, but it could accept an instance of that as a
> parameter
> > instead, much like DoExchange. That would be a breaking change.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
> > >>> Hi David,
> > >>>
> > >>> Going through the ArrowFlight API: got confused a bit on DoGet and
> > >>> DoPut/DoExachange apis: the former one expects FlightDataStream which
> > talks
> > >>> in already serialized message terms while the latter to
> > >>> accept FlightMessageReader/Writer which expect the user to pass in
> > >>> RecordBatches etc. Is there any reason why the DoGet can't have
> > >>> FlightMessageWriter as a return type?
> > >>>
> > >>> Cheers,
> > >>> Gosh
> > >>>
> > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <go...@gmail.com>
> > wrote:
> > >>>
> > >>> > Thanks David!
> > >>> >
> > >>> > I also responded/added more suggestions/questions to the doc. I
> > think it
> > >>> > makes sense to have two sections: one purely protocol oriented and
> > second
> > >>> > API oriented(examples in c++ or in any other language should make
> > the idea
> > >>> > easier to digest).
> > >>> >
> > >>> > Thanks for the reference too!
> > >>> >
> > >>> > Cheers,
> > >>> > Gosh
> > >>> >
> > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org>
> > wrote:
> > >>> >
> > >>> >> Thanks! I've left some initial comments/suggestions to expand it
> in
> > terms
> > >>> >> of the format definitions and not the C++ APIs.
> > >>> >>
> > >>> >> I'll also note something like this was proposed a long time ago -
> > there's
> > >>> >> not very much discussion about it there but for reference:
> > >>> >>
> >
> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> > >>> >> "DoBidirectional"' from 2019-2020). It might be good to address
> why
> > the
> > >>> >> proposed workaround there (union-of-structs) is insufficient for
> > the use
> > >>> >> cases here (and in FlightSQL).
> > >>> >>
> > >>> >> -David
> > >>> >>
> > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> > >>> >> > Ah sorry, comments should work now.
> > >>> >> >
> > >>> >> > Cheers,
> > >>> >> > Gosh
> > >>> >> >
> > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org
> > <mailto:
> > >>> >> lidavidm%40apache.org>> wrote:
> > >>> >> >
> > >>> >> > > Thanks! Will give it a look.
> > >>> >> > >
> > >>> >> > > Would you mind opening it up for comments?
> > >>> >> > >
> > >>> >> > > -David
> > >>> >> > >
> > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com
> > <mailto:
> > >>> >> gosharz%40gmail.com>> wrote:
> > >>> >> > > > Hi folks,
> > >>> >> > > >
> > >>> >> > > > Started putting some thoughts together here:
> > >>> >> > > >
> > >>> >> > >
> > >>> >>
> >
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> > >>> >> > > > Any feedback is welcome!
> > >>> >> > > >
> > >>> >> > > > Cheers,
> > >>> >> > > > Gosh
> > >>> >> > > >
> > >>> >> > >
> > >>> >> >
> > >>> >>
> > >>> >
> > >>>
> > >>
> >
>
>
> --
>
>
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Thanks for chiming in - I've replied in the doc. Scoping it to just schema evolution would be preferable, but I'm not sure if Gosh's usecase requires more flexibility than that or not.

Again, though, given that 1) gRPC recycles a connection, so repeated calls aren't necessarily expensive and 2) encoding tricks like union-of-structs, any solution needs to be weighed against those/we should make sure to document why they aren't sufficient. (For instance, 1) is hampered by the use of L7 load balancers and/or client-side load balancing policies in gRPC and assumes statefulness which is undesirable in general. There's also the eventual desire to have a transport besides gRPC someday.)

-David

On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote:
> Thanks for writing this up! I added a few general comments, but have a
> question on the approach because it's not quite what I was expecting.
> 
> I am slightly concerned that the proposal looks more like support for
> "multiplexing" IPC streams into a single RPC stream rather than support for
> a changing Schema of an otherwise consistently logical stream. gRPC already
> does a good job decoupling RPC streams from one another. I feel that
> throwing that idea into the IPC stream increases client-library
> implementation cost by quite a lot.
> 
> Why is it not good enough to replace the Schema when we see a duplicate?
> This is undoubtedly less work across all client implementations.
> 
> The benefit I see is that you might have two schemas that you swap between
> frequently then you can indicate with a single integer. If that's what you
> want to support I would rather think of them as `schema_id` instead of
> `stream_id` and not give this impression that multiplexing is a goal. As
> you have proposed, it seems that the "done writing for a stream" needs a
> callback notifying the user receiving the stream that a logical subset of
> the flight is complete. Alternatively, if they aren't independent streams
> (to the end-user), we could tell the Arrow layer that a particular schema
> is no longer needed without also needing to communicate further downstream.
> 
> On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org> wrote:
> 
> > Ah to be clear, the API is indeed inconsistent - DoExchange was added some
> > time later (and by its nature returning a FlightDataStream would not have
> > been possible, since it's meant to be able to interleave reading/writing).
> > But really, DoGet is indeed the odd one out in the C++ API and it may be
> > worth correcting. You could also perhaps imagine making a FlightDataStream
> > implementation that accepts a closure and provides it a fake writer, if the
> > API mismatch is hard to work with...
> >
> > That said: this has some benefits, e.g. for a Python service that returns
> > a Table, that means data can be fed into gRPC entirely in C++ without
> > having to bounce into Python for each chunk.
> >
> > Best,
> > David
> >
> > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
> > > Hi David,
> > >
> > > Got you. In fact I was looking at this more from the point of view of
> > consistency of the API in terms of "inputs" and thought DoExchange is kind
> > of a DoGet+ so might make sense to have the same classes being utilized in
> > both places. But again, I might be missing something and I get the point
> > about breaking change.
> > >
> > > Cheers,
> > > Gosh
> > >
> > > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org> wrote:
> > >> __
> > >> It's mostly a quirk of implementation (and just for clarification,
> > they're all nearly identical on the format/protocol level).
> > >>
> > >> DoGet is conceptualized as your application returning a readable stream
> > of batches, instead of your application imperatively writing batches to the
> > client. (This is different than how Flight is implemented in Java.) You
> > would normally not implement FlightDataStream - you would return a
> > RecordBatchStream.
> > >>
> > >> DoGet could not have FlightMessageWriter as a return type as that
> > wouldn't make sense, but it could accept an instance of that as a parameter
> > instead, much like DoExchange. That would be a breaking change.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
> > >>> Hi David,
> > >>>
> > >>> Going through the ArrowFlight API: got confused a bit on DoGet and
> > >>> DoPut/DoExachange apis: the former one expects FlightDataStream which
> > talks
> > >>> in already serialized message terms while the latter to
> > >>> accept FlightMessageReader/Writer which expect the user to pass in
> > >>> RecordBatches etc. Is there any reason why the DoGet can't have
> > >>> FlightMessageWriter as a return type?
> > >>>
> > >>> Cheers,
> > >>> Gosh
> > >>>
> > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <go...@gmail.com>
> > wrote:
> > >>>
> > >>> > Thanks David!
> > >>> >
> > >>> > I also responded/added more suggestions/questions to the doc. I
> > think it
> > >>> > makes sense to have two sections: one purely protocol oriented and
> > second
> > >>> > API oriented(examples in c++ or in any other language should make
> > the idea
> > >>> > easier to digest).
> > >>> >
> > >>> > Thanks for the reference too!
> > >>> >
> > >>> > Cheers,
> > >>> > Gosh
> > >>> >
> > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org>
> > wrote:
> > >>> >
> > >>> >> Thanks! I've left some initial comments/suggestions to expand it in
> > terms
> > >>> >> of the format definitions and not the C++ APIs.
> > >>> >>
> > >>> >> I'll also note something like this was proposed a long time ago -
> > there's
> > >>> >> not very much discussion about it there but for reference:
> > >>> >>
> > https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> > >>> >> "DoBidirectional"' from 2019-2020). It might be good to address why
> > the
> > >>> >> proposed workaround there (union-of-structs) is insufficient for
> > the use
> > >>> >> cases here (and in FlightSQL).
> > >>> >>
> > >>> >> -David
> > >>> >>
> > >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> > >>> >> > Ah sorry, comments should work now.
> > >>> >> >
> > >>> >> > Cheers,
> > >>> >> > Gosh
> > >>> >> >
> > >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org
> > <mailto:
> > >>> >> lidavidm%40apache.org>> wrote:
> > >>> >> >
> > >>> >> > > Thanks! Will give it a look.
> > >>> >> > >
> > >>> >> > > Would you mind opening it up for comments?
> > >>> >> > >
> > >>> >> > > -David
> > >>> >> > >
> > >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com
> > <mailto:
> > >>> >> gosharz%40gmail.com>> wrote:
> > >>> >> > > > Hi folks,
> > >>> >> > > >
> > >>> >> > > > Started putting some thoughts together here:
> > >>> >> > > >
> > >>> >> > >
> > >>> >>
> > https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> > >>> >> > > > Any feedback is welcome!
> > >>> >> > > >
> > >>> >> > > > Cheers,
> > >>> >> > > > Gosh
> > >>> >> > > >
> > >>> >> > >
> > >>> >> >
> > >>> >>
> > >>> >
> > >>>
> > >>
> >
> 
> 
> --
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Nate Bauernfeind <na...@deephaven.io>]

Thanks for writing this up! I added a few general comments, but have a
question on the approach because it's not quite what I was expecting.

I am slightly concerned that the proposal looks more like support for
"multiplexing" IPC streams into a single RPC stream rather than support for
a changing Schema of an otherwise consistently logical stream. gRPC already
does a good job decoupling RPC streams from one another. I feel that
throwing that idea into the IPC stream increases client-library
implementation cost by quite a lot.

Why is it not good enough to replace the Schema when we see a duplicate?
This is undoubtedly less work across all client implementations.

The benefit I see is that you might have two schemas that you swap between
frequently then you can indicate with a single integer. If that's what you
want to support I would rather think of them as `schema_id` instead of
`stream_id` and not give this impression that multiplexing is a goal. As
you have proposed, it seems that the "done writing for a stream" needs a
callback notifying the user receiving the stream that a logical subset of
the flight is complete. Alternatively, if they aren't independent streams
(to the end-user), we could tell the Arrow layer that a particular schema
is no longer needed without also needing to communicate further downstream.

On Wed, Jun 23, 2021 at 1:39 PM David Li <li...@apache.org> wrote:

> Ah to be clear, the API is indeed inconsistent - DoExchange was added some
> time later (and by its nature returning a FlightDataStream would not have
> been possible, since it's meant to be able to interleave reading/writing).
> But really, DoGet is indeed the odd one out in the C++ API and it may be
> worth correcting. You could also perhaps imagine making a FlightDataStream
> implementation that accepts a closure and provides it a fake writer, if the
> API mismatch is hard to work with...
>
> That said: this has some benefits, e.g. for a Python service that returns
> a Table, that means data can be fed into gRPC entirely in C++ without
> having to bounce into Python for each chunk.
>
> Best,
> David
>
> On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
> > Hi David,
> >
> > Got you. In fact I was looking at this more from the point of view of
> consistency of the API in terms of "inputs" and thought DoExchange is kind
> of a DoGet+ so might make sense to have the same classes being utilized in
> both places. But again, I might be missing something and I get the point
> about breaking change.
> >
> > Cheers,
> > Gosh
> >
> > On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org> wrote:
> >> __
> >> It's mostly a quirk of implementation (and just for clarification,
> they're all nearly identical on the format/protocol level).
> >>
> >> DoGet is conceptualized as your application returning a readable stream
> of batches, instead of your application imperatively writing batches to the
> client. (This is different than how Flight is implemented in Java.) You
> would normally not implement FlightDataStream - you would return a
> RecordBatchStream.
> >>
> >> DoGet could not have FlightMessageWriter as a return type as that
> wouldn't make sense, but it could accept an instance of that as a parameter
> instead, much like DoExchange. That would be a breaking change.
> >>
> >> Best,
> >> David
> >>
> >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
> >>> Hi David,
> >>>
> >>> Going through the ArrowFlight API: got confused a bit on DoGet and
> >>> DoPut/DoExachange apis: the former one expects FlightDataStream which
> talks
> >>> in already serialized message terms while the latter to
> >>> accept FlightMessageReader/Writer which expect the user to pass in
> >>> RecordBatches etc. Is there any reason why the DoGet can't have
> >>> FlightMessageWriter as a return type?
> >>>
> >>> Cheers,
> >>> Gosh
> >>>
> >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <go...@gmail.com>
> wrote:
> >>>
> >>> > Thanks David!
> >>> >
> >>> > I also responded/added more suggestions/questions to the doc. I
> think it
> >>> > makes sense to have two sections: one purely protocol oriented and
> second
> >>> > API oriented(examples in c++ or in any other language should make
> the idea
> >>> > easier to digest).
> >>> >
> >>> > Thanks for the reference too!
> >>> >
> >>> > Cheers,
> >>> > Gosh
> >>> >
> >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org>
> wrote:
> >>> >
> >>> >> Thanks! I've left some initial comments/suggestions to expand it in
> terms
> >>> >> of the format definitions and not the C++ APIs.
> >>> >>
> >>> >> I'll also note something like this was proposed a long time ago -
> there's
> >>> >> not very much discussion about it there but for reference:
> >>> >>
> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> >>> >> "DoBidirectional"' from 2019-2020). It might be good to address why
> the
> >>> >> proposed workaround there (union-of-structs) is insufficient for
> the use
> >>> >> cases here (and in FlightSQL).
> >>> >>
> >>> >> -David
> >>> >>
> >>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> >>> >> > Ah sorry, comments should work now.
> >>> >> >
> >>> >> > Cheers,
> >>> >> > Gosh
> >>> >> >
> >>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org
> <mailto:
> >>> >> lidavidm%40apache.org>> wrote:
> >>> >> >
> >>> >> > > Thanks! Will give it a look.
> >>> >> > >
> >>> >> > > Would you mind opening it up for comments?
> >>> >> > >
> >>> >> > > -David
> >>> >> > >
> >>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com
> <mailto:
> >>> >> gosharz%40gmail.com>> wrote:
> >>> >> > > > Hi folks,
> >>> >> > > >
> >>> >> > > > Started putting some thoughts together here:
> >>> >> > > >
> >>> >> > >
> >>> >>
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> >>> >> > > > Any feedback is welcome!
> >>> >> > > >
> >>> >> > > > Cheers,
> >>> >> > > > Gosh
> >>> >> > > >
> >>> >> > >
> >>> >> >
> >>> >>
> >>> >
> >>>
> >>
>


--

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Ah to be clear, the API is indeed inconsistent - DoExchange was added some time later (and by its nature returning a FlightDataStream would not have been possible, since it's meant to be able to interleave reading/writing). But really, DoGet is indeed the odd one out in the C++ API and it may be worth correcting. You could also perhaps imagine making a FlightDataStream implementation that accepts a closure and provides it a fake writer, if the API mismatch is hard to work with...

That said: this has some benefits, e.g. for a Python service that returns a Table, that means data can be fed into gRPC entirely in C++ without having to bounce into Python for each chunk.

Best,
David

On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote:
> Hi David,
> 
> Got you. In fact I was looking at this more from the point of view of consistency of the API in terms of "inputs" and thought DoExchange is kind of a DoGet+ so might make sense to have the same classes being utilized in both places. But again, I might be missing something and I get the point about breaking change.
> 
> Cheers,
> Gosh
> 
> On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org> wrote:
>> __
>> It's mostly a quirk of implementation (and just for clarification, they're all nearly identical on the format/protocol level).
>> 
>> DoGet is conceptualized as your application returning a readable stream of batches, instead of your application imperatively writing batches to the client. (This is different than how Flight is implemented in Java.) You would normally not implement FlightDataStream - you would return a RecordBatchStream.
>> 
>> DoGet could not have FlightMessageWriter as a return type as that wouldn't make sense, but it could accept an instance of that as a parameter instead, much like DoExchange. That would be a breaking change.
>> 
>> Best,
>> David
>> 
>> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
>>> Hi David,
>>> 
>>> Going through the ArrowFlight API: got confused a bit on DoGet and
>>> DoPut/DoExachange apis: the former one expects FlightDataStream which talks
>>> in already serialized message terms while the latter to
>>> accept FlightMessageReader/Writer which expect the user to pass in
>>> RecordBatches etc. Is there any reason why the DoGet can't have
>>> FlightMessageWriter as a return type?
>>> 
>>> Cheers,
>>> Gosh
>>> 
>>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <go...@gmail.com> wrote:
>>> 
>>> > Thanks David!
>>> >
>>> > I also responded/added more suggestions/questions to the doc. I think it
>>> > makes sense to have two sections: one purely protocol oriented and second
>>> > API oriented(examples in c++ or in any other language should make the idea
>>> > easier to digest).
>>> >
>>> > Thanks for the reference too!
>>> >
>>> > Cheers,
>>> > Gosh
>>> >
>>> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org> wrote:
>>> >
>>> >> Thanks! I've left some initial comments/suggestions to expand it in terms
>>> >> of the format definitions and not the C++ APIs.
>>> >>
>>> >> I'll also note something like this was proposed a long time ago - there's
>>> >> not very much discussion about it there but for reference:
>>> >> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
>>> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
>>> >> "DoBidirectional"' from 2019-2020). It might be good to address why the
>>> >> proposed workaround there (union-of-structs) is insufficient for the use
>>> >> cases here (and in FlightSQL).
>>> >>
>>> >> -David
>>> >>
>>> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
>>> >> > Ah sorry, comments should work now.
>>> >> >
>>> >> > Cheers,
>>> >> > Gosh
>>> >> >
>>> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org <mailto:
>>> >> lidavidm%40apache.org>> wrote:
>>> >> >
>>> >> > > Thanks! Will give it a look.
>>> >> > >
>>> >> > > Would you mind opening it up for comments?
>>> >> > >
>>> >> > > -David
>>> >> > >
>>> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com <mailto:
>>> >> gosharz%40gmail.com>> wrote:
>>> >> > > > Hi folks,
>>> >> > > >
>>> >> > > > Started putting some thoughts together here:
>>> >> > > >
>>> >> > >
>>> >> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
>>> >> > > > Any feedback is welcome!
>>> >> > > >
>>> >> > > > Cheers,
>>> >> > > > Gosh
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>> >
>>> 
>> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi David,

Got you. In fact I was looking at this more from the point of view of
consistency of the API in terms of "inputs" and thought DoExchange is kind
of a DoGet+ so might make sense to have the same classes being utilized in
both places. But again, I might be missing something and I get the point
about breaking change.

Cheers,
Gosh

On Wed, Jun 23, 2021 at 2:58 PM David Li <li...@apache.org> wrote:

> It's mostly a quirk of implementation (and just for clarification, they're
> all nearly identical on the format/protocol level).
>
> DoGet is conceptualized as your application returning a readable stream of
> batches, instead of your application imperatively writing batches to the
> client. (This is different than how Flight is implemented in Java.) You
> would normally not implement FlightDataStream - you would return a
> RecordBatchStream.
>
> DoGet could not have FlightMessageWriter as a return type as that wouldn't
> make sense, but it could accept an instance of that as a parameter instead,
> much like DoExchange. That would be a breaking change.
>
> Best,
> David
>
> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
>
> Hi David,
>
> Going through the ArrowFlight API: got confused a bit on DoGet and
> DoPut/DoExachange apis: the former one expects FlightDataStream which talks
> in already serialized message terms while the latter to
> accept FlightMessageReader/Writer which expect the user to pass in
> RecordBatches etc. Is there any reason why the DoGet can't have
> FlightMessageWriter as a return type?
>
> Cheers,
> Gosh
>
> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <go...@gmail.com> wrote:
>
> > Thanks David!
> >
> > I also responded/added more suggestions/questions to the doc. I think it
> > makes sense to have two sections: one purely protocol oriented and second
> > API oriented(examples in c++ or in any other language should make the
> idea
> > easier to digest).
> >
> > Thanks for the reference too!
> >
> > Cheers,
> > Gosh
> >
> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org> wrote:
> >
> >> Thanks! I've left some initial comments/suggestions to expand it in
> terms
> >> of the format definitions and not the C++ APIs.
> >>
> >> I'll also note something like this was proposed a long time ago -
> there's
> >> not very much discussion about it there but for reference:
> >>
> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> >> "DoBidirectional"' from 2019-2020). It might be good to address why the
> >> proposed workaround there (union-of-structs) is insufficient for the use
> >> cases here (and in FlightSQL).
> >>
> >> -David
> >>
> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> >> > Ah sorry, comments should work now.
> >> >
> >> > Cheers,
> >> > Gosh
> >> >
> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org <mailto:
> >> lidavidm%40apache.org>> wrote:
> >> >
> >> > > Thanks! Will give it a look.
> >> > >
> >> > > Would you mind opening it up for comments?
> >> > >
> >> > > -David
> >> > >
> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com <mailto:
> >> gosharz%40gmail.com>> wrote:
> >> > > > Hi folks,
> >> > > >
> >> > > > Started putting some thoughts together here:
> >> > > >
> >> > >
> >>
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> >> > > > Any feedback is welcome!
> >> > > >
> >> > > > Cheers,
> >> > > > Gosh
> >> > > >
> >> > >
> >> >
> >>
> >
>
>
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

It's mostly a quirk of implementation (and just for clarification, they're all nearly identical on the format/protocol level).

DoGet is conceptualized as your application returning a readable stream of batches, instead of your application imperatively writing batches to the client. (This is different than how Flight is implemented in Java.) You would normally not implement FlightDataStream - you would return a RecordBatchStream.

DoGet could not have FlightMessageWriter as a return type as that wouldn't make sense, but it could accept an instance of that as a parameter instead, much like DoExchange. That would be a breaking change.

Best,
David

On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote:
> Hi David,
> 
> Going through the ArrowFlight API: got confused a bit on DoGet and
> DoPut/DoExachange apis: the former one expects FlightDataStream which talks
> in already serialized message terms while the latter to
> accept FlightMessageReader/Writer which expect the user to pass in
> RecordBatches etc. Is there any reason why the DoGet can't have
> FlightMessageWriter as a return type?
> 
> Cheers,
> Gosh
> 
> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <go...@gmail.com> wrote:
> 
> > Thanks David!
> >
> > I also responded/added more suggestions/questions to the doc. I think it
> > makes sense to have two sections: one purely protocol oriented and second
> > API oriented(examples in c++ or in any other language should make the idea
> > easier to digest).
> >
> > Thanks for the reference too!
> >
> > Cheers,
> > Gosh
> >
> > On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org> wrote:
> >
> >> Thanks! I've left some initial comments/suggestions to expand it in terms
> >> of the format definitions and not the C++ APIs.
> >>
> >> I'll also note something like this was proposed a long time ago - there's
> >> not very much discussion about it there but for reference:
> >> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> >> "DoBidirectional"' from 2019-2020). It might be good to address why the
> >> proposed workaround there (union-of-structs) is insufficient for the use
> >> cases here (and in FlightSQL).
> >>
> >> -David
> >>
> >> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> >> > Ah sorry, comments should work now.
> >> >
> >> > Cheers,
> >> > Gosh
> >> >
> >> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org <mailto:
> >> lidavidm%40apache.org>> wrote:
> >> >
> >> > > Thanks! Will give it a look.
> >> > >
> >> > > Would you mind opening it up for comments?
> >> > >
> >> > > -David
> >> > >
> >> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com <mailto:
> >> gosharz%40gmail.com>> wrote:
> >> > > > Hi folks,
> >> > > >
> >> > > > Started putting some thoughts together here:
> >> > > >
> >> > >
> >> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> >> > > > Any feedback is welcome!
> >> > > >
> >> > > > Cheers,
> >> > > > Gosh
> >> > > >
> >> > >
> >> >
> >>
> >
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi David,

Going through the ArrowFlight API: got confused a bit on DoGet and
DoPut/DoExachange apis: the former one expects FlightDataStream which talks
in already serialized message terms while the latter to
accept FlightMessageReader/Writer which expect the user to pass in
RecordBatches etc. Is there any reason why the DoGet can't have
FlightMessageWriter as a return type?

Cheers,
Gosh

On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan <go...@gmail.com> wrote:

> Thanks David!
>
> I also responded/added more suggestions/questions to the doc. I think it
> makes sense to have two sections: one purely protocol oriented and second
> API oriented(examples in c++ or in any other language should make the idea
> easier to digest).
>
> Thanks for the reference too!
>
> Cheers,
> Gosh
>
> On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org> wrote:
>
>> Thanks! I've left some initial comments/suggestions to expand it in terms
>> of the format definitions and not the C++ APIs.
>>
>> I'll also note something like this was proposed a long time ago - there's
>> not very much discussion about it there but for reference:
>> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
>> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
>> "DoBidirectional"' from 2019-2020). It might be good to address why the
>> proposed workaround there (union-of-structs) is insufficient for the use
>> cases here (and in FlightSQL).
>>
>> -David
>>
>> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
>> > Ah sorry, comments should work now.
>> >
>> > Cheers,
>> > Gosh
>> >
>> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org <mailto:
>> lidavidm%40apache.org>> wrote:
>> >
>> > > Thanks! Will give it a look.
>> > >
>> > > Would you mind opening it up for comments?
>> > >
>> > > -David
>> > >
>> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com <mailto:
>> gosharz%40gmail.com>> wrote:
>> > > > Hi folks,
>> > > >
>> > > > Started putting some thoughts together here:
>> > > >
>> > >
>> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
>> > > > Any feedback is welcome!
>> > > >
>> > > > Cheers,
>> > > > Gosh
>> > > >
>> > >
>> >
>>
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Thanks David!

I also responded/added more suggestions/questions to the doc. I think it
makes sense to have two sections: one purely protocol oriented and second
API oriented(examples in c++ or in any other language should make the idea
easier to digest).

Thanks for the reference too!

Cheers,
Gosh

On Mon, Jun 21, 2021 at 4:41 PM David Li <li...@apache.org> wrote:

> Thanks! I've left some initial comments/suggestions to expand it in terms
> of the format definitions and not the C++ APIs.
>
> I'll also note something like this was proposed a long time ago - there's
> not very much discussion about it there but for reference:
> https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E
> (or see the thread '[Discuss][FlightRPC] Extensions to Flight:
> "DoBidirectional"' from 2019-2020). It might be good to address why the
> proposed workaround there (union-of-structs) is insufficient for the use
> cases here (and in FlightSQL).
>
> -David
>
> On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> > Ah sorry, comments should work now.
> >
> > Cheers,
> > Gosh
> >
> > On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org <mailto:
> lidavidm%40apache.org>> wrote:
> >
> > > Thanks! Will give it a look.
> > >
> > > Would you mind opening it up for comments?
> > >
> > > -David
> > >
> > > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com <mailto:
> gosharz%40gmail.com>> wrote:
> > > > Hi folks,
> > > >
> > > > Started putting some thoughts together here:
> > > >
> > >
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> > > > Any feedback is welcome!
> > > >
> > > > Cheers,
> > > > Gosh
> > > >
> > >
> >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Thanks! I've left some initial comments/suggestions to expand it in terms of the format definitions and not the C++ APIs.

I'll also note something like this was proposed a long time ago - there's not very much discussion about it there but for reference: https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E (or see the thread '[Discuss][FlightRPC] Extensions to Flight: "DoBidirectional"' from 2019-2020). It might be good to address why the proposed workaround there (union-of-structs) is insufficient for the use cases here (and in FlightSQL).

-David

On Mon, Jun 21, 2021, at 08:22, Gosh Arzumanyan wrote:
> Ah sorry, comments should work now.
> 
> Cheers,
> Gosh
> 
> On Mon., 21 Jun. 2021, 14:18 David Li, <lidavidm@apache.org <mailto:lidavidm%40apache.org>> wrote:
> 
> > Thanks! Will give it a look.
> >
> > Would you mind opening it up for comments?
> >
> > -David
> >
> > On 2021/06/21 11:56:24, Gosh Arzumanyan <gosharz@gmail.com <mailto:gosharz%40gmail.com>> wrote:
> > > Hi folks,
> > >
> > > Started putting some thoughts together here:
> > >
> > https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> > > Any feedback is welcome!
> > >
> > > Cheers,
> > > Gosh
> > >
> >
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Ah sorry, comments should work now.

Cheers,
Gosh

On Mon., 21 Jun. 2021, 14:18 David Li, <li...@apache.org> wrote:

> Thanks! Will give it a look.
>
> Would you mind opening it up for comments?
>
> -David
>
> On 2021/06/21 11:56:24, Gosh Arzumanyan <go...@gmail.com> wrote:
> > Hi folks,
> >
> > Started putting some thoughts together here:
> >
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> > Any feedback is welcome!
> >
> > Cheers,
> > Gosh
> >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Thanks! Will give it a look. 

Would you mind opening it up for comments?

-David

On 2021/06/21 11:56:24, Gosh Arzumanyan <go...@gmail.com> wrote: 
> Hi folks,
> 
> Started putting some thoughts together here:
> https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
> Any feedback is welcome!
> 
> Cheers,
> Gosh
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi folks,

Started putting some thoughts together here:
https://docs.google.com/document/d/1dIOpKNYwsd9sdChsRBAx37BiJXl_7enpwWkH76n1tOI/edit?usp=sharing
Any feedback is welcome!

Cheers,
Gosh

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi David,

Thanks for poking me on this. I have been thinking it out but have not got
to crafting a doc. Let me put together a rough proposal this weekend.
Afterwards I'll do need your help for bringing it to a reviewable state.

Cheers,
Gosh

On Fri., 18 Jun. 2021, 18:11 David Li, <li...@apache.org> wrote:

> Following up here - Gosh, did you get a chance to put something together?
> Do you need/want help on this? This would also potentially be useful for
> FlightSQL. (See the discussion on GitHub:
> https://github.com/apache/arrow/pull/9368#discussion_r572941765)
>
> Best,
> David
>
> On Fri, Apr 16, 2021, at 10:59, Gosh Arzumanyan wrote:
> > Hi guys!
> >
> > Thanks for the feedback/info.
> > Let me try to put a proposal together. Though I guess I'll need some
> > assistance on crafting it both in terms of the structure of a proposal
> > expected in the Arrow community as well as technical guidance.
> >
> > WIll share a doc with some ideas shortly so that we can start to iterate
> > over it.
> >
> > Cheers,
> > Gosh
> >
> > On Tue, Apr 13, 2021 at 6:55 PM Nate Bauernfeind <
> > natebauernfeind@deephaven.io <mailto:natebauernfeind%40deephaven.io>>
> wrote:
> >
> > > > possibly in coordination with the Deephaven/Barrage team, if they're
> also
> > > still interested
> > >
> > > Good opportunity for me to chime in =). I think we still have interest
> in
> > > this feature. On the other thread, it took a little cajoling, but I've
> come
> > > around to agree with the conclusions of taking a RecordBatch and
> splitting
> > > it up (a set of RecordBatches for added rows followed by a set of
> > > RecordBatches for modifications). In this case I think it's best not to
> > > evolve the schema between added row RecordBatches and modified row
> > > RecordBatches (sending empty buffer nodes and field nodes will be
> > > significantly cheaper). However, the schema evolution would be very
> useful
> > > for when the rpc client changes the set of columns that they are
> subscribed
> > > to (which is relatively rare compared to when the subscribed table
> itself
> > > ticks).
> > >
> > > That said, schema evolution is not yet particularly high in our queue.
> > >
> > > On Tue, Apr 13, 2021 at 9:12 AM David Li <lidavidm@apache.org <mailto:
> lidavidm%40apache.org>> wrote:
> > >
> > > > Thanks for the details. I'll note a few things, but adding schema
> > > > evolution to Flight is reasonable, if you'd like to put together a
> > > > proposal for discussion (possibly in coordination with the
> > > > Deephaven/Barrage team, if they're also still interested).
> > > >
> > > > >    3. Assume that there is a strong reason to query A1,..,AK
> together.
> > > >
> > > > While I don't know the details here, at least with Flight/gRPC, it's
> > > > not necessarily expensive to make several requests to the same
> server,
> > > > as gRPC will consolidate them into the same underlying network
> > > > connection. You could issue one GetFlightInfo request for all streams
> > > > at once, and get back a list of endpoints for each individual
> > > > subquery, which you could then issue separate DoGet requests for.
> > > >
> > > > There's a slight mismatch there in that GetFlightInfo returns a
> > > > FlightInfo, which assumes all endpoints have the same schema. But for
> > > > a specific application, you could ignore that field (nothing in
> Flight
> > > > checks that schema against the actual data).
> > > >
> > > > Of course, if said strong reason is that all the data is really
> > > > retrieved together despite being distinct datasets, then this would
> > > > complicate the server side implementation quite a bit. But it's one
> > > > option.
> > > >
> > > > > A potential way to address this(with the existing tools) could be
> > > having
> > > > a
> > > > > union schema of all fields across all entities(potentially prefixed
> > > with
> > > > > the field name just like in sql joins) and setting the values to NA
> > > which
> > > > > do not belong to an entity.
> > > >
> > > > I had a similar use case in the past, and it was suggested to use
> > > > Arrow's Union type which handles this directly. A Union of Struct
> > > > types essentially lets you have multiple distinct schemas all encoded
> > > > in the same overall table, with explicit information about which
> > > > schema is currently in use. But as you point out this isn't helpful
> if
> > > > you don't know all the schemas up front.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On 2021/04/13 11:21:20, Gosh Arzumanyan <gosharz@gmail.com <mailto:
> gosharz%40gmail.com>> wrote:
> > > > > Hi David,
> > > > >
> > > > > Thanks for sharing the link!
> > > > >
> > > > > Here is how a potential use case might look like:
> > > > >
> > > > >    1. Assume that we have a service S which accepts expressions in
> some
> > > > >    language X.
> > > > >    2. Assume that a typical query to this service requests entities
> > > A_1,
> > > > >    A_2,..,A_K. Each of those entities generates a stream of record
> > > > batches.
> > > > >    Record batches for a single A_I share the same schema, yet
> there is
> > > no
> > > > >    guarantee that schemas are equal across all streams.
> > > > >    3. Assume that there is a strong reason to query A1,..,AK
> together.
> > > > >    4. Service generates record batches(concurrently), tags
> those(e.g.
> > > > with
> > > > >    schema level metadata) and sends them over.
> > > > >
> > > > > A potential way to address this(with the existing tools) could be
> > > having
> > > > a
> > > > > union schema of all fields across all entities(potentially prefixed
> > > with
> > > > > the field name just like in sql joins) and setting the values to NA
> > > which
> > > > > do not belong to an entity. However this solution might not work in
> > > cases
> > > > > where we are not able to construct the unified schema before
> opening
> > > the
> > > > > stream(e.g. in case of changes in the schema for a specific entity
> upon
> > > > > realtime input feeding or an unpredictable generator expression).
> > > > >
> > > > > Cheers,
> > > > > Gosh
> > > > >
> > > > >
> > > > > On Mon., 12 Apr. 2021, 13:45 David Li, <lidavidm@apache.org
> <mailto:lidavidm%40apache.org>> wrote:
> > > > >
> > > > > > Hi Gosh,
> > > > > >
> > > > > > There was indeed a discussion where schema evolution was
> proposed as
> > > a
> > > > > > solution for another use case:
> > > > > >
> > > > > >
> > > >
> > >
> https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E
> > > > > >
> > > > > > I am curious though, what is your use case here?
> > > > > >
> > > > > > Best,
> > > > > > David
> > > > > >
> > > > > > On 2021/04/12 10:49:00, Gosh Arzumanyan <gosharz@gmail.com
> <mailto:gosharz%40gmail.com>> wrote:
> > > > > > > Hi guys, hope you are well!
> > > > > > >
> > > > > > > Judging from the Flight API
> > > > > > > <
> > > > > >
> > > >
> > >
> https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461
> > > > > > >
> > > > > > > and
> > > > > > > from the documentation/examples out there, it seems like data
> > > schema
> > > > is
> > > > > > > supposed to be fixed per stream in ArrowFlight(which is also
> > > aligned
> > > > with
> > > > > > > corresponding IPC stream writers/readers).
> > > > > > > Wondering if the community has evaluated the
> necessity/possibility
> > > of
> > > > > > > supporting schema changes within a single stream(I do recall
> > > seeing a
> > > > > > > discussion on this somewhere but can't find it)?
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Gosh
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Following up here - Gosh, did you get a chance to put something together? Do you need/want help on this? This would also potentially be useful for FlightSQL. (See the discussion on GitHub: https://github.com/apache/arrow/pull/9368#discussion_r572941765)

Best,
David

On Fri, Apr 16, 2021, at 10:59, Gosh Arzumanyan wrote:
> Hi guys!
> 
> Thanks for the feedback/info.
> Let me try to put a proposal together. Though I guess I'll need some
> assistance on crafting it both in terms of the structure of a proposal
> expected in the Arrow community as well as technical guidance.
> 
> WIll share a doc with some ideas shortly so that we can start to iterate
> over it.
> 
> Cheers,
> Gosh
> 
> On Tue, Apr 13, 2021 at 6:55 PM Nate Bauernfeind <
> natebauernfeind@deephaven.io <mailto:natebauernfeind%40deephaven.io>> wrote:
> 
> > > possibly in coordination with the Deephaven/Barrage team, if they're also
> > still interested
> >
> > Good opportunity for me to chime in =). I think we still have interest in
> > this feature. On the other thread, it took a little cajoling, but I've come
> > around to agree with the conclusions of taking a RecordBatch and splitting
> > it up (a set of RecordBatches for added rows followed by a set of
> > RecordBatches for modifications). In this case I think it's best not to
> > evolve the schema between added row RecordBatches and modified row
> > RecordBatches (sending empty buffer nodes and field nodes will be
> > significantly cheaper). However, the schema evolution would be very useful
> > for when the rpc client changes the set of columns that they are subscribed
> > to (which is relatively rare compared to when the subscribed table itself
> > ticks).
> >
> > That said, schema evolution is not yet particularly high in our queue.
> >
> > On Tue, Apr 13, 2021 at 9:12 AM David Li <lidavidm@apache.org <mailto:lidavidm%40apache.org>> wrote:
> >
> > > Thanks for the details. I'll note a few things, but adding schema
> > > evolution to Flight is reasonable, if you'd like to put together a
> > > proposal for discussion (possibly in coordination with the
> > > Deephaven/Barrage team, if they're also still interested).
> > >
> > > >    3. Assume that there is a strong reason to query A1,..,AK together.
> > >
> > > While I don't know the details here, at least with Flight/gRPC, it's
> > > not necessarily expensive to make several requests to the same server,
> > > as gRPC will consolidate them into the same underlying network
> > > connection. You could issue one GetFlightInfo request for all streams
> > > at once, and get back a list of endpoints for each individual
> > > subquery, which you could then issue separate DoGet requests for.
> > >
> > > There's a slight mismatch there in that GetFlightInfo returns a
> > > FlightInfo, which assumes all endpoints have the same schema. But for
> > > a specific application, you could ignore that field (nothing in Flight
> > > checks that schema against the actual data).
> > >
> > > Of course, if said strong reason is that all the data is really
> > > retrieved together despite being distinct datasets, then this would
> > > complicate the server side implementation quite a bit. But it's one
> > > option.
> > >
> > > > A potential way to address this(with the existing tools) could be
> > having
> > > a
> > > > union schema of all fields across all entities(potentially prefixed
> > with
> > > > the field name just like in sql joins) and setting the values to NA
> > which
> > > > do not belong to an entity.
> > >
> > > I had a similar use case in the past, and it was suggested to use
> > > Arrow's Union type which handles this directly. A Union of Struct
> > > types essentially lets you have multiple distinct schemas all encoded
> > > in the same overall table, with explicit information about which
> > > schema is currently in use. But as you point out this isn't helpful if
> > > you don't know all the schemas up front.
> > >
> > > Best,
> > > David
> > >
> > > On 2021/04/13 11:21:20, Gosh Arzumanyan <gosharz@gmail.com <mailto:gosharz%40gmail.com>> wrote:
> > > > Hi David,
> > > >
> > > > Thanks for sharing the link!
> > > >
> > > > Here is how a potential use case might look like:
> > > >
> > > >    1. Assume that we have a service S which accepts expressions in some
> > > >    language X.
> > > >    2. Assume that a typical query to this service requests entities
> > A_1,
> > > >    A_2,..,A_K. Each of those entities generates a stream of record
> > > batches.
> > > >    Record batches for a single A_I share the same schema, yet there is
> > no
> > > >    guarantee that schemas are equal across all streams.
> > > >    3. Assume that there is a strong reason to query A1,..,AK together.
> > > >    4. Service generates record batches(concurrently), tags those(e.g.
> > > with
> > > >    schema level metadata) and sends them over.
> > > >
> > > > A potential way to address this(with the existing tools) could be
> > having
> > > a
> > > > union schema of all fields across all entities(potentially prefixed
> > with
> > > > the field name just like in sql joins) and setting the values to NA
> > which
> > > > do not belong to an entity. However this solution might not work in
> > cases
> > > > where we are not able to construct the unified schema before opening
> > the
> > > > stream(e.g. in case of changes in the schema for a specific entity upon
> > > > realtime input feeding or an unpredictable generator expression).
> > > >
> > > > Cheers,
> > > > Gosh
> > > >
> > > >
> > > > On Mon., 12 Apr. 2021, 13:45 David Li, <lidavidm@apache.org <mailto:lidavidm%40apache.org>> wrote:
> > > >
> > > > > Hi Gosh,
> > > > >
> > > > > There was indeed a discussion where schema evolution was proposed as
> > a
> > > > > solution for another use case:
> > > > >
> > > > >
> > >
> > https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E
> > > > >
> > > > > I am curious though, what is your use case here?
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On 2021/04/12 10:49:00, Gosh Arzumanyan <gosharz@gmail.com <mailto:gosharz%40gmail.com>> wrote:
> > > > > > Hi guys, hope you are well!
> > > > > >
> > > > > > Judging from the Flight API
> > > > > > <
> > > > >
> > >
> > https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461
> > > > > >
> > > > > > and
> > > > > > from the documentation/examples out there, it seems like data
> > schema
> > > is
> > > > > > supposed to be fixed per stream in ArrowFlight(which is also
> > aligned
> > > with
> > > > > > corresponding IPC stream writers/readers).
> > > > > > Wondering if the community has evaluated the necessity/possibility
> > of
> > > > > > supporting schema changes within a single stream(I do recall
> > seeing a
> > > > > > discussion on this somewhere but can't find it)?
> > > > > >
> > > > > > Cheers,
> > > > > > Gosh
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi guys!

Thanks for the feedback/info.
Let me try to put a proposal together. Though I guess I'll need some
assistance on crafting it both in terms of the structure of a proposal
expected in the Arrow community as well as technical guidance.

WIll share a doc with some ideas shortly so that we can start to iterate
over it.

Cheers,
Gosh

On Tue, Apr 13, 2021 at 6:55 PM Nate Bauernfeind <
natebauernfeind@deephaven.io> wrote:

> > possibly in coordination with the Deephaven/Barrage team, if they're also
> still interested
>
> Good opportunity for me to chime in =). I think we still have interest in
> this feature. On the other thread, it took a little cajoling, but I've come
> around to agree with the conclusions of taking a RecordBatch and splitting
> it up (a set of RecordBatches for added rows followed by a set of
> RecordBatches for modifications). In this case I think it's best not to
> evolve the schema between added row RecordBatches and modified row
> RecordBatches (sending empty buffer nodes and field nodes will be
> significantly cheaper). However, the schema evolution would be very useful
> for when the rpc client changes the set of columns that they are subscribed
> to (which is relatively rare compared to when the subscribed table itself
> ticks).
>
> That said, schema evolution is not yet particularly high in our queue.
>
> On Tue, Apr 13, 2021 at 9:12 AM David Li <li...@apache.org> wrote:
>
> > Thanks for the details. I'll note a few things, but adding schema
> > evolution to Flight is reasonable, if you'd like to put together a
> > proposal for discussion (possibly in coordination with the
> > Deephaven/Barrage team, if they're also still interested).
> >
> > >    3. Assume that there is a strong reason to query A1,..,AK together.
> >
> > While I don't know the details here, at least with Flight/gRPC, it's
> > not necessarily expensive to make several requests to the same server,
> > as gRPC will consolidate them into the same underlying network
> > connection. You could issue one GetFlightInfo request for all streams
> > at once, and get back a list of endpoints for each individual
> > subquery, which you could then issue separate DoGet requests for.
> >
> > There's a slight mismatch there in that GetFlightInfo returns a
> > FlightInfo, which assumes all endpoints have the same schema. But for
> > a specific application, you could ignore that field (nothing in Flight
> > checks that schema against the actual data).
> >
> > Of course, if said strong reason is that all the data is really
> > retrieved together despite being distinct datasets, then this would
> > complicate the server side implementation quite a bit. But it's one
> > option.
> >
> > > A potential way to address this(with the existing tools) could be
> having
> > a
> > > union schema of all fields across all entities(potentially prefixed
> with
> > > the field name just like in sql joins) and setting the values to NA
> which
> > > do not belong to an entity.
> >
> > I had a similar use case in the past, and it was suggested to use
> > Arrow's Union type which handles this directly. A Union of Struct
> > types essentially lets you have multiple distinct schemas all encoded
> > in the same overall table, with explicit information about which
> > schema is currently in use. But as you point out this isn't helpful if
> > you don't know all the schemas up front.
> >
> > Best,
> > David
> >
> > On 2021/04/13 11:21:20, Gosh Arzumanyan <go...@gmail.com> wrote:
> > > Hi David,
> > >
> > > Thanks for sharing the link!
> > >
> > > Here is how a potential use case might look like:
> > >
> > >    1. Assume that we have a service S which accepts expressions in some
> > >    language X.
> > >    2. Assume that a typical query to this service requests entities
> A_1,
> > >    A_2,..,A_K. Each of those entities generates a stream of record
> > batches.
> > >    Record batches for a single A_I share the same schema, yet there is
> no
> > >    guarantee that schemas are equal across all streams.
> > >    3. Assume that there is a strong reason to query A1,..,AK together.
> > >    4. Service generates record batches(concurrently), tags those(e.g.
> > with
> > >    schema level metadata) and sends them over.
> > >
> > > A potential way to address this(with the existing tools) could be
> having
> > a
> > > union schema of all fields across all entities(potentially prefixed
> with
> > > the field name just like in sql joins) and setting the values to NA
> which
> > > do not belong to an entity. However this solution might not work in
> cases
> > > where we are not able to construct the unified schema before opening
> the
> > > stream(e.g. in case of changes in the schema for a specific entity upon
> > > realtime input feeding or an unpredictable generator expression).
> > >
> > > Cheers,
> > > Gosh
> > >
> > >
> > > On Mon., 12 Apr. 2021, 13:45 David Li, <li...@apache.org> wrote:
> > >
> > > > Hi Gosh,
> > > >
> > > > There was indeed a discussion where schema evolution was proposed as
> a
> > > > solution for another use case:
> > > >
> > > >
> >
> https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E
> > > >
> > > > I am curious though, what is your use case here?
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On 2021/04/12 10:49:00, Gosh Arzumanyan <go...@gmail.com> wrote:
> > > > > Hi guys, hope you are well!
> > > > >
> > > > > Judging from the Flight API
> > > > > <
> > > >
> >
> https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461
> > > > >
> > > > > and
> > > > > from the documentation/examples out there, it seems like data
> schema
> > is
> > > > > supposed to be fixed per stream in ArrowFlight(which is also
> aligned
> > with
> > > > > corresponding IPC stream writers/readers).
> > > > > Wondering if the community has evaluated the necessity/possibility
> of
> > > > > supporting schema changes within a single stream(I do recall
> seeing a
> > > > > discussion on this somewhere but can't find it)?
> > > > >
> > > > > Cheers,
> > > > > Gosh
> > > > >
> > > >
> > >
> >
>
>
> --
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Nate Bauernfeind <na...@deephaven.io>]

> possibly in coordination with the Deephaven/Barrage team, if they're also
still interested

Good opportunity for me to chime in =). I think we still have interest in
this feature. On the other thread, it took a little cajoling, but I've come
around to agree with the conclusions of taking a RecordBatch and splitting
it up (a set of RecordBatches for added rows followed by a set of
RecordBatches for modifications). In this case I think it's best not to
evolve the schema between added row RecordBatches and modified row
RecordBatches (sending empty buffer nodes and field nodes will be
significantly cheaper). However, the schema evolution would be very useful
for when the rpc client changes the set of columns that they are subscribed
to (which is relatively rare compared to when the subscribed table itself
ticks).

That said, schema evolution is not yet particularly high in our queue.

On Tue, Apr 13, 2021 at 9:12 AM David Li <li...@apache.org> wrote:

> Thanks for the details. I'll note a few things, but adding schema
> evolution to Flight is reasonable, if you'd like to put together a
> proposal for discussion (possibly in coordination with the
> Deephaven/Barrage team, if they're also still interested).
>
> >    3. Assume that there is a strong reason to query A1,..,AK together.
>
> While I don't know the details here, at least with Flight/gRPC, it's
> not necessarily expensive to make several requests to the same server,
> as gRPC will consolidate them into the same underlying network
> connection. You could issue one GetFlightInfo request for all streams
> at once, and get back a list of endpoints for each individual
> subquery, which you could then issue separate DoGet requests for.
>
> There's a slight mismatch there in that GetFlightInfo returns a
> FlightInfo, which assumes all endpoints have the same schema. But for
> a specific application, you could ignore that field (nothing in Flight
> checks that schema against the actual data).
>
> Of course, if said strong reason is that all the data is really
> retrieved together despite being distinct datasets, then this would
> complicate the server side implementation quite a bit. But it's one
> option.
>
> > A potential way to address this(with the existing tools) could be having
> a
> > union schema of all fields across all entities(potentially prefixed with
> > the field name just like in sql joins) and setting the values to NA which
> > do not belong to an entity.
>
> I had a similar use case in the past, and it was suggested to use
> Arrow's Union type which handles this directly. A Union of Struct
> types essentially lets you have multiple distinct schemas all encoded
> in the same overall table, with explicit information about which
> schema is currently in use. But as you point out this isn't helpful if
> you don't know all the schemas up front.
>
> Best,
> David
>
> On 2021/04/13 11:21:20, Gosh Arzumanyan <go...@gmail.com> wrote:
> > Hi David,
> >
> > Thanks for sharing the link!
> >
> > Here is how a potential use case might look like:
> >
> >    1. Assume that we have a service S which accepts expressions in some
> >    language X.
> >    2. Assume that a typical query to this service requests entities A_1,
> >    A_2,..,A_K. Each of those entities generates a stream of record
> batches.
> >    Record batches for a single A_I share the same schema, yet there is no
> >    guarantee that schemas are equal across all streams.
> >    3. Assume that there is a strong reason to query A1,..,AK together.
> >    4. Service generates record batches(concurrently), tags those(e.g.
> with
> >    schema level metadata) and sends them over.
> >
> > A potential way to address this(with the existing tools) could be having
> a
> > union schema of all fields across all entities(potentially prefixed with
> > the field name just like in sql joins) and setting the values to NA which
> > do not belong to an entity. However this solution might not work in cases
> > where we are not able to construct the unified schema before opening the
> > stream(e.g. in case of changes in the schema for a specific entity upon
> > realtime input feeding or an unpredictable generator expression).
> >
> > Cheers,
> > Gosh
> >
> >
> > On Mon., 12 Apr. 2021, 13:45 David Li, <li...@apache.org> wrote:
> >
> > > Hi Gosh,
> > >
> > > There was indeed a discussion where schema evolution was proposed as a
> > > solution for another use case:
> > >
> > >
> https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E
> > >
> > > I am curious though, what is your use case here?
> > >
> > > Best,
> > > David
> > >
> > > On 2021/04/12 10:49:00, Gosh Arzumanyan <go...@gmail.com> wrote:
> > > > Hi guys, hope you are well!
> > > >
> > > > Judging from the Flight API
> > > > <
> > >
> https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461
> > > >
> > > > and
> > > > from the documentation/examples out there, it seems like data schema
> is
> > > > supposed to be fixed per stream in ArrowFlight(which is also aligned
> with
> > > > corresponding IPC stream writers/readers).
> > > > Wondering if the community has evaluated the necessity/possibility of
> > > > supporting schema changes within a single stream(I do recall seeing a
> > > > discussion on this somewhere but can't find it)?
> > > >
> > > > Cheers,
> > > > Gosh
> > > >
> > >
> >
>


--

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Thanks for the details. I'll note a few things, but adding schema
evolution to Flight is reasonable, if you'd like to put together a
proposal for discussion (possibly in coordination with the
Deephaven/Barrage team, if they're also still interested).

>    3. Assume that there is a strong reason to query A1,..,AK together.

While I don't know the details here, at least with Flight/gRPC, it's
not necessarily expensive to make several requests to the same server,
as gRPC will consolidate them into the same underlying network
connection. You could issue one GetFlightInfo request for all streams
at once, and get back a list of endpoints for each individual
subquery, which you could then issue separate DoGet requests for.

There's a slight mismatch there in that GetFlightInfo returns a
FlightInfo, which assumes all endpoints have the same schema. But for
a specific application, you could ignore that field (nothing in Flight
checks that schema against the actual data).

Of course, if said strong reason is that all the data is really
retrieved together despite being distinct datasets, then this would
complicate the server side implementation quite a bit. But it's one
option.

> A potential way to address this(with the existing tools) could be having a
> union schema of all fields across all entities(potentially prefixed with
> the field name just like in sql joins) and setting the values to NA which
> do not belong to an entity.

I had a similar use case in the past, and it was suggested to use
Arrow's Union type which handles this directly. A Union of Struct
types essentially lets you have multiple distinct schemas all encoded
in the same overall table, with explicit information about which
schema is currently in use. But as you point out this isn't helpful if
you don't know all the schemas up front.

Best,
David

On 2021/04/13 11:21:20, Gosh Arzumanyan <go...@gmail.com> wrote: 
> Hi David,
> 
> Thanks for sharing the link!
> 
> Here is how a potential use case might look like:
> 
>    1. Assume that we have a service S which accepts expressions in some
>    language X.
>    2. Assume that a typical query to this service requests entities A_1,
>    A_2,..,A_K. Each of those entities generates a stream of record batches.
>    Record batches for a single A_I share the same schema, yet there is no
>    guarantee that schemas are equal across all streams.
>    3. Assume that there is a strong reason to query A1,..,AK together.
>    4. Service generates record batches(concurrently), tags those(e.g. with
>    schema level metadata) and sends them over.
> 
> A potential way to address this(with the existing tools) could be having a
> union schema of all fields across all entities(potentially prefixed with
> the field name just like in sql joins) and setting the values to NA which
> do not belong to an entity. However this solution might not work in cases
> where we are not able to construct the unified schema before opening the
> stream(e.g. in case of changes in the schema for a specific entity upon
> realtime input feeding or an unpredictable generator expression).
> 
> Cheers,
> Gosh
> 
> 
> On Mon., 12 Apr. 2021, 13:45 David Li, <li...@apache.org> wrote:
> 
> > Hi Gosh,
> >
> > There was indeed a discussion where schema evolution was proposed as a
> > solution for another use case:
> >
> > https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E
> >
> > I am curious though, what is your use case here?
> >
> > Best,
> > David
> >
> > On 2021/04/12 10:49:00, Gosh Arzumanyan <go...@gmail.com> wrote:
> > > Hi guys, hope you are well!
> > >
> > > Judging from the Flight API
> > > <
> > https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461
> > >
> > > and
> > > from the documentation/examples out there, it seems like data schema is
> > > supposed to be fixed per stream in ArrowFlight(which is also aligned with
> > > corresponding IPC stream writers/readers).
> > > Wondering if the community has evaluated the necessity/possibility of
> > > supporting schema changes within a single stream(I do recall seeing a
> > > discussion on this somewhere but can't find it)?
> > >
> > > Cheers,
> > > Gosh
> > >
> >
> 

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[Gosh Arzumanyan <go...@gmail.com>]

Hi David,

Thanks for sharing the link!

Here is how a potential use case might look like:

   1. Assume that we have a service S which accepts expressions in some
   language X.
   2. Assume that a typical query to this service requests entities A_1,
   A_2,..,A_K. Each of those entities generates a stream of record batches.
   Record batches for a single A_I share the same schema, yet there is no
   guarantee that schemas are equal across all streams.
   3. Assume that there is a strong reason to query A1,..,AK together.
   4. Service generates record batches(concurrently), tags those(e.g. with
   schema level metadata) and sends them over.

A potential way to address this(with the existing tools) could be having a
union schema of all fields across all entities(potentially prefixed with
the field name just like in sql joins) and setting the values to NA which
do not belong to an entity. However this solution might not work in cases
where we are not able to construct the unified schema before opening the
stream(e.g. in case of changes in the schema for a specific entity upon
realtime input feeding or an unpredictable generator expression).

Cheers,
Gosh


On Mon., 12 Apr. 2021, 13:45 David Li, <li...@apache.org> wrote:

> Hi Gosh,
>
> There was indeed a discussion where schema evolution was proposed as a
> solution for another use case:
>
> https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E
>
> I am curious though, what is your use case here?
>
> Best,
> David
>
> On 2021/04/12 10:49:00, Gosh Arzumanyan <go...@gmail.com> wrote:
> > Hi guys, hope you are well!
> >
> > Judging from the Flight API
> > <
> https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461
> >
> > and
> > from the documentation/examples out there, it seems like data schema is
> > supposed to be fixed per stream in ArrowFlight(which is also aligned with
> > corresponding IPC stream writers/readers).
> > Wondering if the community has evaluated the necessity/possibility of
> > supporting schema changes within a single stream(I do recall seeing a
> > discussion on this somewhere but can't find it)?
> >
> > Cheers,
> > Gosh
> >
>

Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams

[David Li <li...@apache.org>]

Hi Gosh,

There was indeed a discussion where schema evolution was proposed as a solution for another use case:
https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E

I am curious though, what is your use case here?

Best,
David

On 2021/04/12 10:49:00, Gosh Arzumanyan <go...@gmail.com> wrote: 
> Hi guys, hope you are well!
> 
> Judging from the Flight API
> <https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461>
> and
> from the documentation/examples out there, it seems like data schema is
> supposed to be fixed per stream in ArrowFlight(which is also aligned with
> corresponding IPC stream writers/readers).
> Wondering if the community has evaluated the necessity/possibility of
> supporting schema changes within a single stream(I do recall seeing a
> discussion on this somewhere but can't find it)?
> 
> Cheers,
> Gosh
>