You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@paimon.apache.org by zouxxyy <zo...@gmail.com> on 2024/01/12 07:00:49 UTC

[DISCUSS] PIP-14: Paimon statistics

Hi, Paimon Devs, I’d like to start a discussion about PIP-14[1].

Table statistics describe the data distribution characteristics of a table. 
Common statistics include the number of rows, table size, column statistics and more. 
They are very important for DBMS, especially when executing query plans and optimizing query performance.
This PIP further expand on the existing statistics of Paimon to support more statistical information.

Look forward to your question and suggestions. 

Best, zouxxyy

[1] https://cwiki.apache.org/confluence/x/HYokEQ


Re: [DISCUSS] PIP-14: Paimon statistics

Posted by Jingsong Li <ji...@gmail.com>.
Thanks zouxxy for starting this discussion.

The design looks good to me overall.

Left some comments:

> Statistics in snapshot (global stats + col stats) with some prunning strategies > stats calculated in real-time from splits (only including numRows and totalSize)

But also need to re-calculate partition col stats?

> FileStoreCommit.writeStats

FileStoreCommit.commitStatistics

> Long snapshotId in Stats

Why nullable? Should it be long?

Best,
Jingsong

On Fri, Jan 12, 2024 at 3:01 PM zouxxyy <zo...@gmail.com> wrote:
>
> Hi, Paimon Devs, I’d like to start a discussion about PIP-14[1].
>
> Table statistics describe the data distribution characteristics of a table.
> Common statistics include the number of rows, table size, column statistics and more.
> They are very important for DBMS, especially when executing query plans and optimizing query performance.
> This PIP further expand on the existing statistics of Paimon to support more statistical information.
>
> Look forward to your question and suggestions.
>
> Best, zouxxyy
>
> [1] https://cwiki.apache.org/confluence/x/HYokEQ
>