[DISCUSS] More public interfaces for internal data structures

[Jingsong Li <ji...@gmail.com>]

Hi everyone,

The source sink for the Table/SQL has now been completely switched to
a new API (DynamicTableFactory), which requires connector developers
to prioritize the use of internal data structures. For example,
SourceProvider and SinkProvider.

There are complex connectors that have many requirements for
manipulating data structures:
- Use BinaryRowData, including BinaryRowWriter, The binary format can
help the connector store data more efficiently, such as how to cache
rows in memory
- Projection for RowData
- Comparator for internal structure.
- Equal for internal structure.
- toString for internal structure.

Without these things, the connector will be very inconvenient to
implement, such as even printing out data is very troublesome.

Can we provide these utils with public interfaces?

Best,
Jingsong

Re: [DISCUSS] More public interfaces for internal data structures

[Jark Wu <im...@gmail.com>]

Thank Jingsong for starting this discussion.

I think it's reasonable to add them to public APIs
which can help build connectors easier.
Looking forward to a FLIP to finalize the APIs.

Best,
Jark

On Tue, 26 Apr 2022 at 14:03, Jingsong Li <ji...@gmail.com> wrote:

> Hi everyone,
>
> The source sink for the Table/SQL has now been completely switched to
> a new API (DynamicTableFactory), which requires connector developers
> to prioritize the use of internal data structures. For example,
> SourceProvider and SinkProvider.
>
> There are complex connectors that have many requirements for
> manipulating data structures:
> - Use BinaryRowData, including BinaryRowWriter, The binary format can
> help the connector store data more efficiently, such as how to cache
> rows in memory
> - Projection for RowData
> - Comparator for internal structure.
> - Equal for internal structure.
> - toString for internal structure.
>
> Without these things, the connector will be very inconvenient to
> implement, such as even printing out data is very troublesome.
>
> Can we provide these utils with public interfaces?
>
> Best,
> Jingsong
>