[DISCUSSION] IEP-92: Binary Tuple Format

[Aleksey Demakov <ad...@gmail.com>]

Hi all,

I drafted a proposal for a new data format for Apache Ignite 3.0. For
lack of a better name it is called Binary Tuple Format [1]. The idea
is to have a common representation of rows and other similar data
whenever that data is transferred and handled in binary form by
different database modules. For instance, a storage engine could
produce a batch of rows for the SQL engine, the SQL engine might do
some operations over it like filtering, projection, joining and then
pass the result further for another round of operations on a different
node, and then the final result will be delivered to the client. It
seems beneficial to use the same format everywhere as long as it is
possible. Please note that the final SQL query result may have very
little to do with any actual table schema stored in the database. So
here, in contrast with e.g. IEP-54 [2], a tuple does not necessarily
equate to a table row and has a broader meaning.

I would ask the community for a review. Any feedback would be greatly
appreciated.

Regards,
Aleksey

[1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-92%3A+Binary+Tuple+Format
[2] https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach

Re: [DISCUSSION] IEP-92: Binary Tuple Format

[Pavel Tupitsyn <pt...@apache.org>]

Aleksey,

I fully support the idea of a common tuple format to avoid conversions when
passing data around (servers, clients).
The IEP looks good to me.

> Integer Field Compression
Agree with this, we can even extend the idea to float/double fields.

>  size of table elements might be 1, 2, 4, or 8 bytes depending on the
header flags
Right now we are limited by the max size of byte[] in Java
(Integer.MAX_VALUE) in many places, so 8-byte offsets are probably not
needed.

On Thu, Jun 16, 2022 at 5:04 PM Aleksey Demakov <ad...@gmail.com> wrote:

> Hi all,
>
> I drafted a proposal for a new data format for Apache Ignite 3.0. For
> lack of a better name it is called Binary Tuple Format [1]. The idea
> is to have a common representation of rows and other similar data
> whenever that data is transferred and handled in binary form by
> different database modules. For instance, a storage engine could
> produce a batch of rows for the SQL engine, the SQL engine might do
> some operations over it like filtering, projection, joining and then
> pass the result further for another round of operations on a different
> node, and then the final result will be delivered to the client. It
> seems beneficial to use the same format everywhere as long as it is
> possible. Please note that the final SQL query result may have very
> little to do with any actual table schema stored in the database. So
> here, in contrast with e.g. IEP-54 [2], a tuple does not necessarily
> equate to a table row and has a broader meaning.
>
> I would ask the community for a review. Any feedback would be greatly
> appreciated.
>
> Regards,
> Aleksey
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-92%3A+Binary+Tuple+Format
> [2]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
>

Re: [DISCUSSION] IEP-92: Binary Tuple Format

[Aleksey Demakov <ad...@gmail.com>]

Hi all,

I updated IEP-92 [1] with information on binary representation of
different data types.

Regards,
Aleksey

[1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-92%3A+Binary+Tuple+Format