[FlightRPC] Flight fallback mechanism.

[Vinay Kesarwani <vn...@gmail.com>]

Flight is great way to share data among processes, and holding shared data
in-memory.

I couldn't find any api or design to handle fall back mechanism in case
data is not fitting in memory.

Cases:
1- Once memory buffer is nearing full, data should spill over to disk.
2- Spilling over disk or memory mapped file.
3- Should it be .arrow file or feather format on Disk.
4- Should it be compressed? any design suggestion?

Can we have a feasibility check and performance impact in terms of
benchmark #.

Re: [FlightRPC] Flight fallback mechanism.

[Wes McKinney <we...@gmail.com>]

hi Vinay,

I think these are application-level concerns -- Flight provides an
efficient means to transport datasets on a network. The result of a
DoGet or DoPut provides an iterator of record batches which generally
may not be that big. How you handle the chunks of data in memory (or
put them on disk, etc.) seems out of scope for Flight itself.

- Wes

On Mon, Dec 23, 2019 at 3:27 AM Vinay Kesarwani <vn...@gmail.com> wrote:
>
> Flight is great way to share data among processes, and holding shared data
> in-memory.
>
> I couldn't find any api or design to handle fall back mechanism in case
> data is not fitting in memory.
>
> Cases:
> 1- Once memory buffer is nearing full, data should spill over to disk.
> 2- Spilling over disk or memory mapped file.
> 3- Should it be .arrow file or feather format on Disk.
> 4- Should it be compressed? any design suggestion?
>
> Can we have a feasibility check and performance impact in terms of
> benchmark #.