You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Felipe Oliveira Carvalho <fe...@gmail.com> on 2024/07/01 14:58:44 UTC

Re: [DISCUSS] Statistics through the C data interface

Hi,

You can promise that well-known int32 statistic keys won't ever be higher
than a certain value (2^18) [1] like TCP IP ports (well-known ports in [0,
2^10)) but for non-standard statistics from open-source products the key=0
combined with string label is the way to go, otherwise collisions would be
inevitable and everyone would hate us for having integer keys.

This is not a very weird proposal from my part because integer keys
representing labels are common in most low-level standardized C APIs (e.g.
linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
map all these keys to strings, but with them we keep the "C Data Interface"
low-level and portable as it should be.

--
Felipe

[1] 2^16 is too small. For instance, OpenGL constants can't be enums
because C limits enum to 2^16 and that is *not enough*.

On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:

> Hi,
>
> Here is an updated summary so far:
>
> ----
> Use cases:
>
> * Optimize query plan: e.g. JOIN for DuckDB
>
> Out of scope:
>
> * Transmit statistics through not the C data interface
>   Examples:
>   * Transmit statistics through Apache Arrow IPC file
>   * Transmit statistics through Apache Arrow Flight
> * Multi-column statistics
> * Constraints information
> * Indexes information
>
> Discussing approach:
>
> Standardize Apache Arrow schema for statistics and transmit
> statistics via separated API call that uses the C data
> interface.
>
> This also works for per-batch statistics.
>
> Candidate schema:
>
> map<
>   // The column index or null if the statistics refer to whole table or
> batch.
>   column: int32,
>   // Statistics key is int32.
>   // Different keys are assigned for exact value and
>   // approximate value.
>   map<int32, dense_union<...needed types based on stat kinds in the
> keys...>>
> >
>
> Discussions:
>
> 1. Can we use int32 for statistic keys?
>    Should we use utf8 (or dictionary<int32, utf8>) for
>    statistic keys?
> 2. Hot to support non-standard (vendor-specific)
>    statistic keys?
> ----
>
> Here is my idea:
>
> 1. We can use int32 for statistic keys.
> 2. We can reserve a specific range for non-standard
>    statistic keys. Prerequisites of this:
>    * There is no use case to merge some statistics for
>      the same data.
>    * We can't merge statistics for different data.
>
> If the prerequisites aren't satisfied:
>
> 1. We should use utf8 (or dictionary<int32, utf8>) for
>    statistic keys?
> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>    standard statistic keys or use prefix such as
>    "vendor1:"/"vendor1." for non-standard statistic keys.
>
> Here is Felipe's idea:
> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>
> 1. We can use int32 for statistic keys.
> 2. We can use the special statistic key + a string identifier
>    for non-standard statistic keys.
>
>
> What do you think about this?
>
>
> Thanks,
> --
> kou
>
> In <20...@clear-code.com>
>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
> 2024 18:27:27 +0900 (JST),
>   Sutou Kouhei <ko...@clear-code.com> wrote:
>
> > Hi,
> >
> > Thanks for sharing your comments. Here is a summary so far:
> >
> > ----
> >
> > Use cases:
> >
> > * Optimize query plan: e.g. JOIN for DuckDB
> >
> > Out of scope:
> >
> > * Transmit statistics through not the C data interface
> >   Examples:
> >   * Transmit statistics through Apache Arrow IPC file
> >   * Transmit statistics through Apache Arrow Flight
> >
> > Candidate approaches:
> >
> > 1. Pass statistics (encoded as an Apache Arrow data) via
> >    ArrowSchema metadata
> >    * This embeds statistics address into metadata
> >    * It's for avoiding using Apache Arrow IPC format with
> >      the C data interface
> > 2. Embed statistics (encoded as an Apache Arrow data) into
> >    ArrowSchema metadata
> >    * This adds statistics to metadata in Apache Arrow IPC
> >      format
> > 3. Embed statistics (encoded as JSON) into ArrowArray
> >    metadata
> > 4. Standardize Apache Arrow schema for statistics and
> >    transmit statistics via separated API call that uses the
> >    C data interface
> > 5. Use ADBC
> >
> > ----
> >
> > I think that 4. is the best approach in these candidates.
> >
> > 1. Embedding statistics address is tricky.
> > 2. Consumers need to parse Apache Arrow IPC format data.
> >    (The C data interface consumers may not have the
> >    feature.)
> > 3. This will work but 4. is more generic.
> > 5. ADBC is too large to use only for statistics.
> >
> > What do you think about this?
> >
> >
> > If we select 4., we need to standardize Apache Arrow schema
> > for statistics. How about the following schema?
> >
> > ----
> > Metadata:
> >
> > | Name                       | Value | Comments |
> > |----------------------------|-------|--------- |
> > | ARROW::statistics::version | 1.0.0 | (1)      |
> >
> > (1) This follows semantic versioning.
> >
> > Fields:
> >
> > | Name           | Type                  | Comments |
> > |----------------|-----------------------| -------- |
> > | column         | utf8                  | (2)      |
> > | key            | utf8 not null         | (3)      |
> > | value          | VALUE_SCHEMA not null |          |
> > | is_approximate | bool not null         | (4)      |
> >
> > (2) If null, then the statistic applies to the entire table.
> >     It's for "row_count".
> > (3) We'll provide pre-defined keys such as "max", "min",
> >     "byte_width" and "distinct_count" but users can also use
> >     application specific keys.
> > (4) If true, then the value is approximate or best-effort.
> >
> > VALUE_SCHEMA is a dense union with members:
> >
> > | Name    | Type    |
> > |---------|---------|
> > | int64   | int64   |
> > | uint64  | uint64  |
> > | float64 | float64 |
> > | binary  | binary  |
> >
> > If a column is an int32 column, it uses int64 for
> > "max"/"min". We don't provide all types here. Users should
> > use a compatible type (int64 for a int32 column) instead.
> > ----
> >
> >
> > Thanks,
> > --
> > kou
> >
> >
> > In <20...@clear-code.com>
> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
> 2024 11:37:08 +0900 (JST),
> >   Sutou Kouhei <ko...@clear-code.com> wrote:
> >
> >> Hi,
> >>
> >> We're discussing how to provide statistics through the C
> >> data interface at:
> >> https://github.com/apache/arrow/issues/38837
> >>
> >> If you're interested in this feature, could you share your
> >> comments?
> >>
> >>
> >> Motivation:
> >>
> >> We can interchange Apache Arrow data by the C data interface
> >> in the same process. For example, we can pass Apache Arrow
> >> data read by Apache Arrow C++ (provider) to DuckDB
> >> (consumer) through the C data interface.
> >>
> >> A provider may know Apache Arrow data statistics. For
> >> example, a provider can know statistics when it reads Apache
> >> Parquet data because Apache Parquet may provide statistics.
> >>
> >> But a consumer can't know statistics that are known by a
> >> producer. Because there isn't a standard way to provide
> >> statistics through the C data interface. If a consumer can
> >> know statistics, it can process Apache Arrow data faster
> >> based on statistics.
> >>
> >>
> >> Proposal:
> >>
> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
> >>
> >> How about providing statistics as a metadata in ArrowSchema?
> >>
> >> We reserve "ARROW" namespace for internal Apache Arrow use:
> >>
> >>
> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
> >>
> >>> The ARROW pattern is a reserved namespace for internal
> >>> Arrow use in the custom_metadata fields. For example,
> >>> ARROW:extension:name.
> >>
> >> So we can use "ARROW:statistics" for the metadata key.
> >>
> >> We can represent statistics as a ArrowArray like ADBC does.
> >>
> >> Here is an example ArrowSchema that is for a record batch
> >> that has "int32 column1" and "string column2":
> >>
> >> ArrowSchema {
> >>   .format = "+siu",
> >>   .metadata = {
> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
> as row count */
> >>   },
> >>   .children = {
> >>     ArrowSchema {
> >>       .name = "column1",
> >>       .format = "i",
> >>       .metadata = {
> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
> such as count distinct */
> >>       },
> >>     },
> >>     ArrowSchema {
> >>       .name = "column2",
> >>       .format = "u",
> >>       .metadata = {
> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
> such as count distinct */
> >>       },
> >>     },
> >>   },
> >> }
> >>
> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
> >> => ArrowArray*' is a base 10 string of the address of the
> >> ArrowArray. Because we can use only string for metadata
> >> value. You can't release the statistics ArrowArray*. (Its
> >> release is a no-op function.) It follows
> >>
> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
> >> semantics. (The base ArrowSchema owns statistics
> >> ArrowArray*.)
> >>
> >>
> >> ArrowArray* for statistics use the following schema:
> >>
> >> | Field Name     | Field Type                       | Comments |
> >> |----------------|----------------------------------| -------- |
> >> | key            | string not null                  | (1)      |
> >> | value          | `VALUE_SCHEMA` not null          |          |
> >> | is_approximate | bool not null                    | (2)      |
> >>
> >> 1. We'll provide pre-defined keys such as "max", "min",
> >>    "byte_width" and "distinct_count" but users can also use
> >>    application specific keys.
> >>
> >> 2. If true, then the value is approximate or best-effort.
> >>
> >> VALUE_SCHEMA is a dense union with members:
> >>
> >> | Field Name | Field Type                       | Comments |
> >> |------------|----------------------------------| -------- |
> >> | int64      | int64                            |          |
> >> | uint64     | uint64                           |          |
> >> | float64    | float64                          |          |
> >> | value      | The same type of the ArrowSchema | (3)      |
> >> |            | that is belonged to.             |          |
> >>
> >> 3. If the ArrowSchema's type is string, this type is also string.
> >>
> >>    TODO: Is "value" good name? If we refer it from the
> >>    top-level statistics schema, we need to use
> >>    "value.value". It's a bit strange...
> >>
> >>
> >> What do you think about this proposal? Could you share your
> >> comments?
> >>
> >>
> >> Thanks,
> >> --
> >> kou
>

Re: [DISCUSS] Statistics through the C data interface

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

It seems that how to build the statistics schema isn't so
complex. I'll proceed with this approach.


Thanks,
-- 
kou

In <20...@clear-code.com>
  "Re: [DISCUSS] Statistics through the C data interface" on Wed, 16 Oct 2024 15:08:37 +0900 (JST),
  Sutou Kouhei <ko...@clear-code.com> wrote:

> Hi,
> 
> I'm implementing C++ producer for statistics array:
> https://github.com/apache/arrow/pull/44252
> 
> Our discussed statistics schema is compact:
> https://github.com/apache/arrow/pull/43553/files#diff-f3758fb6986ea8d24bb2e13c2feb625b68bbd6b93b3fbafd3e2a03dcdc7ba263R77-R145
> 
> But it may be a bit complex to build.
> 
> What do you think about this?
> 
> 
> Thanks,
> -- 
> kou
> 
> In <20...@clear-code.com>
>   "Re: [DISCUSS] Statistics through the C data interface" on Mon, 05 Aug 2024 18:33:31 +0900 (JST),
>   Sutou Kouhei <ko...@clear-code.com> wrote:
> 
>> Hi,
>> 
>> I've opened a PR for documentation:
>> https://github.com/apache/arrow/pull/43553
>> 
>> The "Example" section isn't written yet but suggestions are
>> very welcome.
>> 
>> Thanks,
>> -- 
>> kou
>> 
>> In <20...@clear-code.com>
>>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 25 Jul 2024 14:35:18 +0900 (JST),
>>   Sutou Kouhei <ko...@clear-code.com> wrote:
>> 
>>> Hi,
>>> 
>>> If nobody objects using utf8 or dictionary<int32, utf8> for
>>> statistics key, let's use dictionary<int32, utf8>. Because
>>> dictionary<int32, utf8> will be more effective than utf8
>>> when there are many columns.
>>> 
>>> I'll start writing a documentation for this and implementing
>>> this for C++ next week. I'll share a PR for them when I
>>> complete them. We can start a vote for this after we review
>>> the PR.
>>> 
>>> 
>>> Thanks,
>>> -- 
>>> kou
>>> 
>>> In <20...@clear-code.com>
>>>   "Re: [DISCUSS] Statistics through the C data interface" on Fri, 12 Jul 2024 15:15:36 +0900 (JST),
>>>   Sutou Kouhei <ko...@clear-code.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>>>> map<struct<int32, utf8>,
>>>>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>>>>
>>>>> 
>>>>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>>>>> unions gracefully, this could be:
>>>>> 
>>>>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>>>>> in the keys...>>
>>>>> 
>>>>> X is either sparse or dense.
>>>>> 
>>>>> A possible alternative is to use a custom struct instead of map and reduce
>>>>> the levels of nesting:
>>>>> 
>>>>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
>>>> 
>>>> Thanks for clarifying your suggestion.
>>>> 
>>>> If we need utf8 for non-standard statistics, I think that
>>>> map<utf8, ...> or map<dictionary<int32, utf8>, ...> is
>>>> better as Antoine said. Because they are simpler than
>>>> int32+utf8.
>>>> 
>>>> 
>>>> Thanks,
>>>> -- 
>>>> kou
>>>> 
>>>> In <CA...@mail.gmail.com>
>>>>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 11 Jul 2024 14:17:46 -0300,
>>>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>>>> 
>>>>> On Thu, Jul 11, 2024 at 5:04 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>> 
>>>>>> Hi,
>>>>>>
>>>>>> >            for non-standard statistics from open-source products the
>>>>>> key=0
>>>>>> > combined with string label is the way to go
>>>>>>
>>>>>> Where do we store the string label?
>>>>>>
>>>>>> I think that we're considering the following schema:
>>>>>>
>>>>>> >> map<
>>>>>> >>   // The column index or null if the statistics refer to whole table or
>>>>>> batch.
>>>>>> >>   column: int32,
>>>>>> >>   // Statistics key is int32.
>>>>>> >>   // Different keys are assigned for exact value and
>>>>>> >>   // approximate value.
>>>>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>>>>> keys...>>
>>>>>> >> >
>>>>>>
>>>>>> Are you considering the following schema for key=0 case?
>>>>>>
>>>>>> map<struct<int32, utf8>,
>>>>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>>>>
>>>>> 
>>>>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>>>>> unions gracefully, this could be:
>>>>> 
>>>>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>>>>> in the keys...>>
>>>>> 
>>>>> X is either sparse or dense.
>>>>> 
>>>>> A possible alternative is to use a custom struct instead of map and reduce
>>>>> the levels of nesting:
>>>>> 
>>>>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
>>>>> 
>>>>> --
>>>>> Felipe
>>>>> 
>>>>> 
>>>>>>
>>>>>> Thanks,
>>>>>> --
>>>>>> kou
>>>>>>
>>>>>> In <CA...@mail.gmail.com>
>>>>>>   "Re: [DISCUSS] Statistics through the C data interface" on Mon, 1 Jul
>>>>>> 2024 11:58:44 -0300,
>>>>>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > You can promise that well-known int32 statistic keys won't ever be higher
>>>>>> > than a certain value (2^18) [1] like TCP IP ports (well-known ports in
>>>>>> [0,
>>>>>> > 2^10)) but for non-standard statistics from open-source products the
>>>>>> key=0
>>>>>> > combined with string label is the way to go, otherwise collisions would
>>>>>> be
>>>>>> > inevitable and everyone would hate us for having integer keys.
>>>>>> >
>>>>>> > This is not a very weird proposal from my part because integer keys
>>>>>> > representing labels are common in most low-level standardized C APIs
>>>>>> (e.g.
>>>>>> > linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
>>>>>> > map all these keys to strings, but with them we keep the "C Data
>>>>>> Interface"
>>>>>> > low-level and portable as it should be.
>>>>>> >
>>>>>> > --
>>>>>> > Felipe
>>>>>> >
>>>>>> > [1] 2^16 is too small. For instance, OpenGL constants can't be enums
>>>>>> > because C limits enum to 2^16 and that is *not enough*.
>>>>>> >
>>>>>> > On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>>> >
>>>>>> >> Hi,
>>>>>> >>
>>>>>> >> Here is an updated summary so far:
>>>>>> >>
>>>>>> >> ----
>>>>>> >> Use cases:
>>>>>> >>
>>>>>> >> * Optimize query plan: e.g. JOIN for DuckDB
>>>>>> >>
>>>>>> >> Out of scope:
>>>>>> >>
>>>>>> >> * Transmit statistics through not the C data interface
>>>>>> >>   Examples:
>>>>>> >>   * Transmit statistics through Apache Arrow IPC file
>>>>>> >>   * Transmit statistics through Apache Arrow Flight
>>>>>> >> * Multi-column statistics
>>>>>> >> * Constraints information
>>>>>> >> * Indexes information
>>>>>> >>
>>>>>> >> Discussing approach:
>>>>>> >>
>>>>>> >> Standardize Apache Arrow schema for statistics and transmit
>>>>>> >> statistics via separated API call that uses the C data
>>>>>> >> interface.
>>>>>> >>
>>>>>> >> This also works for per-batch statistics.
>>>>>> >>
>>>>>> >> Candidate schema:
>>>>>> >>
>>>>>> >> map<
>>>>>> >>   // The column index or null if the statistics refer to whole table or
>>>>>> >> batch.
>>>>>> >>   column: int32,
>>>>>> >>   // Statistics key is int32.
>>>>>> >>   // Different keys are assigned for exact value and
>>>>>> >>   // approximate value.
>>>>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>>>>> >> keys...>>
>>>>>> >> >
>>>>>> >>
>>>>>> >> Discussions:
>>>>>> >>
>>>>>> >> 1. Can we use int32 for statistic keys?
>>>>>> >>    Should we use utf8 (or dictionary<int32, utf8>) for
>>>>>> >>    statistic keys?
>>>>>> >> 2. Hot to support non-standard (vendor-specific)
>>>>>> >>    statistic keys?
>>>>>> >> ----
>>>>>> >>
>>>>>> >> Here is my idea:
>>>>>> >>
>>>>>> >> 1. We can use int32 for statistic keys.
>>>>>> >> 2. We can reserve a specific range for non-standard
>>>>>> >>    statistic keys. Prerequisites of this:
>>>>>> >>    * There is no use case to merge some statistics for
>>>>>> >>      the same data.
>>>>>> >>    * We can't merge statistics for different data.
>>>>>> >>
>>>>>> >> If the prerequisites aren't satisfied:
>>>>>> >>
>>>>>> >> 1. We should use utf8 (or dictionary<int32, utf8>) for
>>>>>> >>    statistic keys?
>>>>>> >> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>>>>>> >>    standard statistic keys or use prefix such as
>>>>>> >>    "vendor1:"/"vendor1." for non-standard statistic keys.
>>>>>> >>
>>>>>> >> Here is Felipe's idea:
>>>>>> >> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>>>>>> >>
>>>>>> >> 1. We can use int32 for statistic keys.
>>>>>> >> 2. We can use the special statistic key + a string identifier
>>>>>> >>    for non-standard statistic keys.
>>>>>> >>
>>>>>> >>
>>>>>> >> What do you think about this?
>>>>>> >>
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> --
>>>>>> >> kou
>>>>>> >>
>>>>>> >> In <20...@clear-code.com>
>>>>>> >>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
>>>>>> >> 2024 18:27:27 +0900 (JST),
>>>>>> >>   Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>>> >>
>>>>>> >> > Hi,
>>>>>> >> >
>>>>>> >> > Thanks for sharing your comments. Here is a summary so far:
>>>>>> >> >
>>>>>> >> > ----
>>>>>> >> >
>>>>>> >> > Use cases:
>>>>>> >> >
>>>>>> >> > * Optimize query plan: e.g. JOIN for DuckDB
>>>>>> >> >
>>>>>> >> > Out of scope:
>>>>>> >> >
>>>>>> >> > * Transmit statistics through not the C data interface
>>>>>> >> >   Examples:
>>>>>> >> >   * Transmit statistics through Apache Arrow IPC file
>>>>>> >> >   * Transmit statistics through Apache Arrow Flight
>>>>>> >> >
>>>>>> >> > Candidate approaches:
>>>>>> >> >
>>>>>> >> > 1. Pass statistics (encoded as an Apache Arrow data) via
>>>>>> >> >    ArrowSchema metadata
>>>>>> >> >    * This embeds statistics address into metadata
>>>>>> >> >    * It's for avoiding using Apache Arrow IPC format with
>>>>>> >> >      the C data interface
>>>>>> >> > 2. Embed statistics (encoded as an Apache Arrow data) into
>>>>>> >> >    ArrowSchema metadata
>>>>>> >> >    * This adds statistics to metadata in Apache Arrow IPC
>>>>>> >> >      format
>>>>>> >> > 3. Embed statistics (encoded as JSON) into ArrowArray
>>>>>> >> >    metadata
>>>>>> >> > 4. Standardize Apache Arrow schema for statistics and
>>>>>> >> >    transmit statistics via separated API call that uses the
>>>>>> >> >    C data interface
>>>>>> >> > 5. Use ADBC
>>>>>> >> >
>>>>>> >> > ----
>>>>>> >> >
>>>>>> >> > I think that 4. is the best approach in these candidates.
>>>>>> >> >
>>>>>> >> > 1. Embedding statistics address is tricky.
>>>>>> >> > 2. Consumers need to parse Apache Arrow IPC format data.
>>>>>> >> >    (The C data interface consumers may not have the
>>>>>> >> >    feature.)
>>>>>> >> > 3. This will work but 4. is more generic.
>>>>>> >> > 5. ADBC is too large to use only for statistics.
>>>>>> >> >
>>>>>> >> > What do you think about this?
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > If we select 4., we need to standardize Apache Arrow schema
>>>>>> >> > for statistics. How about the following schema?
>>>>>> >> >
>>>>>> >> > ----
>>>>>> >> > Metadata:
>>>>>> >> >
>>>>>> >> > | Name                       | Value | Comments |
>>>>>> >> > |----------------------------|-------|--------- |
>>>>>> >> > | ARROW::statistics::version | 1.0.0 | (1)      |
>>>>>> >> >
>>>>>> >> > (1) This follows semantic versioning.
>>>>>> >> >
>>>>>> >> > Fields:
>>>>>> >> >
>>>>>> >> > | Name           | Type                  | Comments |
>>>>>> >> > |----------------|-----------------------| -------- |
>>>>>> >> > | column         | utf8                  | (2)      |
>>>>>> >> > | key            | utf8 not null         | (3)      |
>>>>>> >> > | value          | VALUE_SCHEMA not null |          |
>>>>>> >> > | is_approximate | bool not null         | (4)      |
>>>>>> >> >
>>>>>> >> > (2) If null, then the statistic applies to the entire table.
>>>>>> >> >     It's for "row_count".
>>>>>> >> > (3) We'll provide pre-defined keys such as "max", "min",
>>>>>> >> >     "byte_width" and "distinct_count" but users can also use
>>>>>> >> >     application specific keys.
>>>>>> >> > (4) If true, then the value is approximate or best-effort.
>>>>>> >> >
>>>>>> >> > VALUE_SCHEMA is a dense union with members:
>>>>>> >> >
>>>>>> >> > | Name    | Type    |
>>>>>> >> > |---------|---------|
>>>>>> >> > | int64   | int64   |
>>>>>> >> > | uint64  | uint64  |
>>>>>> >> > | float64 | float64 |
>>>>>> >> > | binary  | binary  |
>>>>>> >> >
>>>>>> >> > If a column is an int32 column, it uses int64 for
>>>>>> >> > "max"/"min". We don't provide all types here. Users should
>>>>>> >> > use a compatible type (int64 for a int32 column) instead.
>>>>>> >> > ----
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > Thanks,
>>>>>> >> > --
>>>>>> >> > kou
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > In <20...@clear-code.com>
>>>>>> >> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
>>>>>> >> 2024 11:37:08 +0900 (JST),
>>>>>> >> >   Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>>> >> >
>>>>>> >> >> Hi,
>>>>>> >> >>
>>>>>> >> >> We're discussing how to provide statistics through the C
>>>>>> >> >> data interface at:
>>>>>> >> >> https://github.com/apache/arrow/issues/38837
>>>>>> >> >>
>>>>>> >> >> If you're interested in this feature, could you share your
>>>>>> >> >> comments?
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> Motivation:
>>>>>> >> >>
>>>>>> >> >> We can interchange Apache Arrow data by the C data interface
>>>>>> >> >> in the same process. For example, we can pass Apache Arrow
>>>>>> >> >> data read by Apache Arrow C++ (provider) to DuckDB
>>>>>> >> >> (consumer) through the C data interface.
>>>>>> >> >>
>>>>>> >> >> A provider may know Apache Arrow data statistics. For
>>>>>> >> >> example, a provider can know statistics when it reads Apache
>>>>>> >> >> Parquet data because Apache Parquet may provide statistics.
>>>>>> >> >>
>>>>>> >> >> But a consumer can't know statistics that are known by a
>>>>>> >> >> producer. Because there isn't a standard way to provide
>>>>>> >> >> statistics through the C data interface. If a consumer can
>>>>>> >> >> know statistics, it can process Apache Arrow data faster
>>>>>> >> >> based on statistics.
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> Proposal:
>>>>>> >> >>
>>>>>> >> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
>>>>>> >> >>
>>>>>> >> >> How about providing statistics as a metadata in ArrowSchema?
>>>>>> >> >>
>>>>>> >> >> We reserve "ARROW" namespace for internal Apache Arrow use:
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >>
>>>>>> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
>>>>>> >> >>
>>>>>> >> >>> The ARROW pattern is a reserved namespace for internal
>>>>>> >> >>> Arrow use in the custom_metadata fields. For example,
>>>>>> >> >>> ARROW:extension:name.
>>>>>> >> >>
>>>>>> >> >> So we can use "ARROW:statistics" for the metadata key.
>>>>>> >> >>
>>>>>> >> >> We can represent statistics as a ArrowArray like ADBC does.
>>>>>> >> >>
>>>>>> >> >> Here is an example ArrowSchema that is for a record batch
>>>>>> >> >> that has "int32 column1" and "string column2":
>>>>>> >> >>
>>>>>> >> >> ArrowSchema {
>>>>>> >> >>   .format = "+siu",
>>>>>> >> >>   .metadata = {
>>>>>> >> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
>>>>>> >> as row count */
>>>>>> >> >>   },
>>>>>> >> >>   .children = {
>>>>>> >> >>     ArrowSchema {
>>>>>> >> >>       .name = "column1",
>>>>>> >> >>       .format = "i",
>>>>>> >> >>       .metadata = {
>>>>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>>>>> >> such as count distinct */
>>>>>> >> >>       },
>>>>>> >> >>     },
>>>>>> >> >>     ArrowSchema {
>>>>>> >> >>       .name = "column2",
>>>>>> >> >>       .format = "u",
>>>>>> >> >>       .metadata = {
>>>>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>>>>> >> such as count distinct */
>>>>>> >> >>       },
>>>>>> >> >>     },
>>>>>> >> >>   },
>>>>>> >> >> }
>>>>>> >> >>
>>>>>> >> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
>>>>>> >> >> => ArrowArray*' is a base 10 string of the address of the
>>>>>> >> >> ArrowArray. Because we can use only string for metadata
>>>>>> >> >> value. You can't release the statistics ArrowArray*. (Its
>>>>>> >> >> release is a no-op function.) It follows
>>>>>> >> >>
>>>>>> >>
>>>>>> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
>>>>>> >> >> semantics. (The base ArrowSchema owns statistics
>>>>>> >> >> ArrowArray*.)
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> ArrowArray* for statistics use the following schema:
>>>>>> >> >>
>>>>>> >> >> | Field Name     | Field Type                       | Comments |
>>>>>> >> >> |----------------|----------------------------------| -------- |
>>>>>> >> >> | key            | string not null                  | (1)      |
>>>>>> >> >> | value          | `VALUE_SCHEMA` not null          |          |
>>>>>> >> >> | is_approximate | bool not null                    | (2)      |
>>>>>> >> >>
>>>>>> >> >> 1. We'll provide pre-defined keys such as "max", "min",
>>>>>> >> >>    "byte_width" and "distinct_count" but users can also use
>>>>>> >> >>    application specific keys.
>>>>>> >> >>
>>>>>> >> >> 2. If true, then the value is approximate or best-effort.
>>>>>> >> >>
>>>>>> >> >> VALUE_SCHEMA is a dense union with members:
>>>>>> >> >>
>>>>>> >> >> | Field Name | Field Type                       | Comments |
>>>>>> >> >> |------------|----------------------------------| -------- |
>>>>>> >> >> | int64      | int64                            |          |
>>>>>> >> >> | uint64     | uint64                           |          |
>>>>>> >> >> | float64    | float64                          |          |
>>>>>> >> >> | value      | The same type of the ArrowSchema | (3)      |
>>>>>> >> >> |            | that is belonged to.             |          |
>>>>>> >> >>
>>>>>> >> >> 3. If the ArrowSchema's type is string, this type is also string.
>>>>>> >> >>
>>>>>> >> >>    TODO: Is "value" good name? If we refer it from the
>>>>>> >> >>    top-level statistics schema, we need to use
>>>>>> >> >>    "value.value". It's a bit strange...
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> What do you think about this proposal? Could you share your
>>>>>> >> >> comments?
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >> Thanks,
>>>>>> >> >> --
>>>>>> >> >> kou
>>>>>> >>
>>>>>>

Re: [DISCUSS] Statistics through the C data interface

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

I'm implementing C++ producer for statistics array:
https://github.com/apache/arrow/pull/44252

Our discussed statistics schema is compact:
https://github.com/apache/arrow/pull/43553/files#diff-f3758fb6986ea8d24bb2e13c2feb625b68bbd6b93b3fbafd3e2a03dcdc7ba263R77-R145

But it may be a bit complex to build.

What do you think about this?


Thanks,
-- 
kou

In <20...@clear-code.com>
  "Re: [DISCUSS] Statistics through the C data interface" on Mon, 05 Aug 2024 18:33:31 +0900 (JST),
  Sutou Kouhei <ko...@clear-code.com> wrote:

> Hi,
> 
> I've opened a PR for documentation:
> https://github.com/apache/arrow/pull/43553
> 
> The "Example" section isn't written yet but suggestions are
> very welcome.
> 
> Thanks,
> -- 
> kou
> 
> In <20...@clear-code.com>
>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 25 Jul 2024 14:35:18 +0900 (JST),
>   Sutou Kouhei <ko...@clear-code.com> wrote:
> 
>> Hi,
>> 
>> If nobody objects using utf8 or dictionary<int32, utf8> for
>> statistics key, let's use dictionary<int32, utf8>. Because
>> dictionary<int32, utf8> will be more effective than utf8
>> when there are many columns.
>> 
>> I'll start writing a documentation for this and implementing
>> this for C++ next week. I'll share a PR for them when I
>> complete them. We can start a vote for this after we review
>> the PR.
>> 
>> 
>> Thanks,
>> -- 
>> kou
>> 
>> In <20...@clear-code.com>
>>   "Re: [DISCUSS] Statistics through the C data interface" on Fri, 12 Jul 2024 15:15:36 +0900 (JST),
>>   Sutou Kouhei <ko...@clear-code.com> wrote:
>> 
>>> Hi,
>>> 
>>>>> map<struct<int32, utf8>,
>>>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>>>
>>>> 
>>>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>>>> unions gracefully, this could be:
>>>> 
>>>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>>>> in the keys...>>
>>>> 
>>>> X is either sparse or dense.
>>>> 
>>>> A possible alternative is to use a custom struct instead of map and reduce
>>>> the levels of nesting:
>>>> 
>>>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
>>> 
>>> Thanks for clarifying your suggestion.
>>> 
>>> If we need utf8 for non-standard statistics, I think that
>>> map<utf8, ...> or map<dictionary<int32, utf8>, ...> is
>>> better as Antoine said. Because they are simpler than
>>> int32+utf8.
>>> 
>>> 
>>> Thanks,
>>> -- 
>>> kou
>>> 
>>> In <CA...@mail.gmail.com>
>>>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 11 Jul 2024 14:17:46 -0300,
>>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>>> 
>>>> On Thu, Jul 11, 2024 at 5:04 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>>>> 
>>>>> Hi,
>>>>>
>>>>> >            for non-standard statistics from open-source products the
>>>>> key=0
>>>>> > combined with string label is the way to go
>>>>>
>>>>> Where do we store the string label?
>>>>>
>>>>> I think that we're considering the following schema:
>>>>>
>>>>> >> map<
>>>>> >>   // The column index or null if the statistics refer to whole table or
>>>>> batch.
>>>>> >>   column: int32,
>>>>> >>   // Statistics key is int32.
>>>>> >>   // Different keys are assigned for exact value and
>>>>> >>   // approximate value.
>>>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>>>> keys...>>
>>>>> >> >
>>>>>
>>>>> Are you considering the following schema for key=0 case?
>>>>>
>>>>> map<struct<int32, utf8>,
>>>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>>>
>>>> 
>>>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>>>> unions gracefully, this could be:
>>>> 
>>>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>>>> in the keys...>>
>>>> 
>>>> X is either sparse or dense.
>>>> 
>>>> A possible alternative is to use a custom struct instead of map and reduce
>>>> the levels of nesting:
>>>> 
>>>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
>>>> 
>>>> --
>>>> Felipe
>>>> 
>>>> 
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> kou
>>>>>
>>>>> In <CA...@mail.gmail.com>
>>>>>   "Re: [DISCUSS] Statistics through the C data interface" on Mon, 1 Jul
>>>>> 2024 11:58:44 -0300,
>>>>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > You can promise that well-known int32 statistic keys won't ever be higher
>>>>> > than a certain value (2^18) [1] like TCP IP ports (well-known ports in
>>>>> [0,
>>>>> > 2^10)) but for non-standard statistics from open-source products the
>>>>> key=0
>>>>> > combined with string label is the way to go, otherwise collisions would
>>>>> be
>>>>> > inevitable and everyone would hate us for having integer keys.
>>>>> >
>>>>> > This is not a very weird proposal from my part because integer keys
>>>>> > representing labels are common in most low-level standardized C APIs
>>>>> (e.g.
>>>>> > linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
>>>>> > map all these keys to strings, but with them we keep the "C Data
>>>>> Interface"
>>>>> > low-level and portable as it should be.
>>>>> >
>>>>> > --
>>>>> > Felipe
>>>>> >
>>>>> > [1] 2^16 is too small. For instance, OpenGL constants can't be enums
>>>>> > because C limits enum to 2^16 and that is *not enough*.
>>>>> >
>>>>> > On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>> >
>>>>> >> Hi,
>>>>> >>
>>>>> >> Here is an updated summary so far:
>>>>> >>
>>>>> >> ----
>>>>> >> Use cases:
>>>>> >>
>>>>> >> * Optimize query plan: e.g. JOIN for DuckDB
>>>>> >>
>>>>> >> Out of scope:
>>>>> >>
>>>>> >> * Transmit statistics through not the C data interface
>>>>> >>   Examples:
>>>>> >>   * Transmit statistics through Apache Arrow IPC file
>>>>> >>   * Transmit statistics through Apache Arrow Flight
>>>>> >> * Multi-column statistics
>>>>> >> * Constraints information
>>>>> >> * Indexes information
>>>>> >>
>>>>> >> Discussing approach:
>>>>> >>
>>>>> >> Standardize Apache Arrow schema for statistics and transmit
>>>>> >> statistics via separated API call that uses the C data
>>>>> >> interface.
>>>>> >>
>>>>> >> This also works for per-batch statistics.
>>>>> >>
>>>>> >> Candidate schema:
>>>>> >>
>>>>> >> map<
>>>>> >>   // The column index or null if the statistics refer to whole table or
>>>>> >> batch.
>>>>> >>   column: int32,
>>>>> >>   // Statistics key is int32.
>>>>> >>   // Different keys are assigned for exact value and
>>>>> >>   // approximate value.
>>>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>>>> >> keys...>>
>>>>> >> >
>>>>> >>
>>>>> >> Discussions:
>>>>> >>
>>>>> >> 1. Can we use int32 for statistic keys?
>>>>> >>    Should we use utf8 (or dictionary<int32, utf8>) for
>>>>> >>    statistic keys?
>>>>> >> 2. Hot to support non-standard (vendor-specific)
>>>>> >>    statistic keys?
>>>>> >> ----
>>>>> >>
>>>>> >> Here is my idea:
>>>>> >>
>>>>> >> 1. We can use int32 for statistic keys.
>>>>> >> 2. We can reserve a specific range for non-standard
>>>>> >>    statistic keys. Prerequisites of this:
>>>>> >>    * There is no use case to merge some statistics for
>>>>> >>      the same data.
>>>>> >>    * We can't merge statistics for different data.
>>>>> >>
>>>>> >> If the prerequisites aren't satisfied:
>>>>> >>
>>>>> >> 1. We should use utf8 (or dictionary<int32, utf8>) for
>>>>> >>    statistic keys?
>>>>> >> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>>>>> >>    standard statistic keys or use prefix such as
>>>>> >>    "vendor1:"/"vendor1." for non-standard statistic keys.
>>>>> >>
>>>>> >> Here is Felipe's idea:
>>>>> >> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>>>>> >>
>>>>> >> 1. We can use int32 for statistic keys.
>>>>> >> 2. We can use the special statistic key + a string identifier
>>>>> >>    for non-standard statistic keys.
>>>>> >>
>>>>> >>
>>>>> >> What do you think about this?
>>>>> >>
>>>>> >>
>>>>> >> Thanks,
>>>>> >> --
>>>>> >> kou
>>>>> >>
>>>>> >> In <20...@clear-code.com>
>>>>> >>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
>>>>> >> 2024 18:27:27 +0900 (JST),
>>>>> >>   Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>> >>
>>>>> >> > Hi,
>>>>> >> >
>>>>> >> > Thanks for sharing your comments. Here is a summary so far:
>>>>> >> >
>>>>> >> > ----
>>>>> >> >
>>>>> >> > Use cases:
>>>>> >> >
>>>>> >> > * Optimize query plan: e.g. JOIN for DuckDB
>>>>> >> >
>>>>> >> > Out of scope:
>>>>> >> >
>>>>> >> > * Transmit statistics through not the C data interface
>>>>> >> >   Examples:
>>>>> >> >   * Transmit statistics through Apache Arrow IPC file
>>>>> >> >   * Transmit statistics through Apache Arrow Flight
>>>>> >> >
>>>>> >> > Candidate approaches:
>>>>> >> >
>>>>> >> > 1. Pass statistics (encoded as an Apache Arrow data) via
>>>>> >> >    ArrowSchema metadata
>>>>> >> >    * This embeds statistics address into metadata
>>>>> >> >    * It's for avoiding using Apache Arrow IPC format with
>>>>> >> >      the C data interface
>>>>> >> > 2. Embed statistics (encoded as an Apache Arrow data) into
>>>>> >> >    ArrowSchema metadata
>>>>> >> >    * This adds statistics to metadata in Apache Arrow IPC
>>>>> >> >      format
>>>>> >> > 3. Embed statistics (encoded as JSON) into ArrowArray
>>>>> >> >    metadata
>>>>> >> > 4. Standardize Apache Arrow schema for statistics and
>>>>> >> >    transmit statistics via separated API call that uses the
>>>>> >> >    C data interface
>>>>> >> > 5. Use ADBC
>>>>> >> >
>>>>> >> > ----
>>>>> >> >
>>>>> >> > I think that 4. is the best approach in these candidates.
>>>>> >> >
>>>>> >> > 1. Embedding statistics address is tricky.
>>>>> >> > 2. Consumers need to parse Apache Arrow IPC format data.
>>>>> >> >    (The C data interface consumers may not have the
>>>>> >> >    feature.)
>>>>> >> > 3. This will work but 4. is more generic.
>>>>> >> > 5. ADBC is too large to use only for statistics.
>>>>> >> >
>>>>> >> > What do you think about this?
>>>>> >> >
>>>>> >> >
>>>>> >> > If we select 4., we need to standardize Apache Arrow schema
>>>>> >> > for statistics. How about the following schema?
>>>>> >> >
>>>>> >> > ----
>>>>> >> > Metadata:
>>>>> >> >
>>>>> >> > | Name                       | Value | Comments |
>>>>> >> > |----------------------------|-------|--------- |
>>>>> >> > | ARROW::statistics::version | 1.0.0 | (1)      |
>>>>> >> >
>>>>> >> > (1) This follows semantic versioning.
>>>>> >> >
>>>>> >> > Fields:
>>>>> >> >
>>>>> >> > | Name           | Type                  | Comments |
>>>>> >> > |----------------|-----------------------| -------- |
>>>>> >> > | column         | utf8                  | (2)      |
>>>>> >> > | key            | utf8 not null         | (3)      |
>>>>> >> > | value          | VALUE_SCHEMA not null |          |
>>>>> >> > | is_approximate | bool not null         | (4)      |
>>>>> >> >
>>>>> >> > (2) If null, then the statistic applies to the entire table.
>>>>> >> >     It's for "row_count".
>>>>> >> > (3) We'll provide pre-defined keys such as "max", "min",
>>>>> >> >     "byte_width" and "distinct_count" but users can also use
>>>>> >> >     application specific keys.
>>>>> >> > (4) If true, then the value is approximate or best-effort.
>>>>> >> >
>>>>> >> > VALUE_SCHEMA is a dense union with members:
>>>>> >> >
>>>>> >> > | Name    | Type    |
>>>>> >> > |---------|---------|
>>>>> >> > | int64   | int64   |
>>>>> >> > | uint64  | uint64  |
>>>>> >> > | float64 | float64 |
>>>>> >> > | binary  | binary  |
>>>>> >> >
>>>>> >> > If a column is an int32 column, it uses int64 for
>>>>> >> > "max"/"min". We don't provide all types here. Users should
>>>>> >> > use a compatible type (int64 for a int32 column) instead.
>>>>> >> > ----
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> > --
>>>>> >> > kou
>>>>> >> >
>>>>> >> >
>>>>> >> > In <20...@clear-code.com>
>>>>> >> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
>>>>> >> 2024 11:37:08 +0900 (JST),
>>>>> >> >   Sutou Kouhei <ko...@clear-code.com> wrote:
>>>>> >> >
>>>>> >> >> Hi,
>>>>> >> >>
>>>>> >> >> We're discussing how to provide statistics through the C
>>>>> >> >> data interface at:
>>>>> >> >> https://github.com/apache/arrow/issues/38837
>>>>> >> >>
>>>>> >> >> If you're interested in this feature, could you share your
>>>>> >> >> comments?
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Motivation:
>>>>> >> >>
>>>>> >> >> We can interchange Apache Arrow data by the C data interface
>>>>> >> >> in the same process. For example, we can pass Apache Arrow
>>>>> >> >> data read by Apache Arrow C++ (provider) to DuckDB
>>>>> >> >> (consumer) through the C data interface.
>>>>> >> >>
>>>>> >> >> A provider may know Apache Arrow data statistics. For
>>>>> >> >> example, a provider can know statistics when it reads Apache
>>>>> >> >> Parquet data because Apache Parquet may provide statistics.
>>>>> >> >>
>>>>> >> >> But a consumer can't know statistics that are known by a
>>>>> >> >> producer. Because there isn't a standard way to provide
>>>>> >> >> statistics through the C data interface. If a consumer can
>>>>> >> >> know statistics, it can process Apache Arrow data faster
>>>>> >> >> based on statistics.
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Proposal:
>>>>> >> >>
>>>>> >> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
>>>>> >> >>
>>>>> >> >> How about providing statistics as a metadata in ArrowSchema?
>>>>> >> >>
>>>>> >> >> We reserve "ARROW" namespace for internal Apache Arrow use:
>>>>> >> >>
>>>>> >> >>
>>>>> >>
>>>>> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
>>>>> >> >>
>>>>> >> >>> The ARROW pattern is a reserved namespace for internal
>>>>> >> >>> Arrow use in the custom_metadata fields. For example,
>>>>> >> >>> ARROW:extension:name.
>>>>> >> >>
>>>>> >> >> So we can use "ARROW:statistics" for the metadata key.
>>>>> >> >>
>>>>> >> >> We can represent statistics as a ArrowArray like ADBC does.
>>>>> >> >>
>>>>> >> >> Here is an example ArrowSchema that is for a record batch
>>>>> >> >> that has "int32 column1" and "string column2":
>>>>> >> >>
>>>>> >> >> ArrowSchema {
>>>>> >> >>   .format = "+siu",
>>>>> >> >>   .metadata = {
>>>>> >> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
>>>>> >> as row count */
>>>>> >> >>   },
>>>>> >> >>   .children = {
>>>>> >> >>     ArrowSchema {
>>>>> >> >>       .name = "column1",
>>>>> >> >>       .format = "i",
>>>>> >> >>       .metadata = {
>>>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>>>> >> such as count distinct */
>>>>> >> >>       },
>>>>> >> >>     },
>>>>> >> >>     ArrowSchema {
>>>>> >> >>       .name = "column2",
>>>>> >> >>       .format = "u",
>>>>> >> >>       .metadata = {
>>>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>>>> >> such as count distinct */
>>>>> >> >>       },
>>>>> >> >>     },
>>>>> >> >>   },
>>>>> >> >> }
>>>>> >> >>
>>>>> >> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
>>>>> >> >> => ArrowArray*' is a base 10 string of the address of the
>>>>> >> >> ArrowArray. Because we can use only string for metadata
>>>>> >> >> value. You can't release the statistics ArrowArray*. (Its
>>>>> >> >> release is a no-op function.) It follows
>>>>> >> >>
>>>>> >>
>>>>> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
>>>>> >> >> semantics. (The base ArrowSchema owns statistics
>>>>> >> >> ArrowArray*.)
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> ArrowArray* for statistics use the following schema:
>>>>> >> >>
>>>>> >> >> | Field Name     | Field Type                       | Comments |
>>>>> >> >> |----------------|----------------------------------| -------- |
>>>>> >> >> | key            | string not null                  | (1)      |
>>>>> >> >> | value          | `VALUE_SCHEMA` not null          |          |
>>>>> >> >> | is_approximate | bool not null                    | (2)      |
>>>>> >> >>
>>>>> >> >> 1. We'll provide pre-defined keys such as "max", "min",
>>>>> >> >>    "byte_width" and "distinct_count" but users can also use
>>>>> >> >>    application specific keys.
>>>>> >> >>
>>>>> >> >> 2. If true, then the value is approximate or best-effort.
>>>>> >> >>
>>>>> >> >> VALUE_SCHEMA is a dense union with members:
>>>>> >> >>
>>>>> >> >> | Field Name | Field Type                       | Comments |
>>>>> >> >> |------------|----------------------------------| -------- |
>>>>> >> >> | int64      | int64                            |          |
>>>>> >> >> | uint64     | uint64                           |          |
>>>>> >> >> | float64    | float64                          |          |
>>>>> >> >> | value      | The same type of the ArrowSchema | (3)      |
>>>>> >> >> |            | that is belonged to.             |          |
>>>>> >> >>
>>>>> >> >> 3. If the ArrowSchema's type is string, this type is also string.
>>>>> >> >>
>>>>> >> >>    TODO: Is "value" good name? If we refer it from the
>>>>> >> >>    top-level statistics schema, we need to use
>>>>> >> >>    "value.value". It's a bit strange...
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> What do you think about this proposal? Could you share your
>>>>> >> >> comments?
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> Thanks,
>>>>> >> >> --
>>>>> >> >> kou
>>>>> >>
>>>>>

Re: [DISCUSS] Statistics through the C data interface

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

I've opened a PR for documentation:
https://github.com/apache/arrow/pull/43553

The "Example" section isn't written yet but suggestions are
very welcome.

Thanks,
-- 
kou

In <20...@clear-code.com>
  "Re: [DISCUSS] Statistics through the C data interface" on Thu, 25 Jul 2024 14:35:18 +0900 (JST),
  Sutou Kouhei <ko...@clear-code.com> wrote:

> Hi,
> 
> If nobody objects using utf8 or dictionary<int32, utf8> for
> statistics key, let's use dictionary<int32, utf8>. Because
> dictionary<int32, utf8> will be more effective than utf8
> when there are many columns.
> 
> I'll start writing a documentation for this and implementing
> this for C++ next week. I'll share a PR for them when I
> complete them. We can start a vote for this after we review
> the PR.
> 
> 
> Thanks,
> -- 
> kou
> 
> In <20...@clear-code.com>
>   "Re: [DISCUSS] Statistics through the C data interface" on Fri, 12 Jul 2024 15:15:36 +0900 (JST),
>   Sutou Kouhei <ko...@clear-code.com> wrote:
> 
>> Hi,
>> 
>>>> map<struct<int32, utf8>,
>>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>>
>>> 
>>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>>> unions gracefully, this could be:
>>> 
>>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>>> in the keys...>>
>>> 
>>> X is either sparse or dense.
>>> 
>>> A possible alternative is to use a custom struct instead of map and reduce
>>> the levels of nesting:
>>> 
>>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
>> 
>> Thanks for clarifying your suggestion.
>> 
>> If we need utf8 for non-standard statistics, I think that
>> map<utf8, ...> or map<dictionary<int32, utf8>, ...> is
>> better as Antoine said. Because they are simpler than
>> int32+utf8.
>> 
>> 
>> Thanks,
>> -- 
>> kou
>> 
>> In <CA...@mail.gmail.com>
>>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 11 Jul 2024 14:17:46 -0300,
>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>> 
>>> On Thu, Jul 11, 2024 at 5:04 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>>> 
>>>> Hi,
>>>>
>>>> >            for non-standard statistics from open-source products the
>>>> key=0
>>>> > combined with string label is the way to go
>>>>
>>>> Where do we store the string label?
>>>>
>>>> I think that we're considering the following schema:
>>>>
>>>> >> map<
>>>> >>   // The column index or null if the statistics refer to whole table or
>>>> batch.
>>>> >>   column: int32,
>>>> >>   // Statistics key is int32.
>>>> >>   // Different keys are assigned for exact value and
>>>> >>   // approximate value.
>>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>>> keys...>>
>>>> >> >
>>>>
>>>> Are you considering the following schema for key=0 case?
>>>>
>>>> map<struct<int32, utf8>,
>>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>>
>>> 
>>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>>> unions gracefully, this could be:
>>> 
>>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>>> in the keys...>>
>>> 
>>> X is either sparse or dense.
>>> 
>>> A possible alternative is to use a custom struct instead of map and reduce
>>> the levels of nesting:
>>> 
>>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
>>> 
>>> --
>>> Felipe
>>> 
>>> 
>>>>
>>>> Thanks,
>>>> --
>>>> kou
>>>>
>>>> In <CA...@mail.gmail.com>
>>>>   "Re: [DISCUSS] Statistics through the C data interface" on Mon, 1 Jul
>>>> 2024 11:58:44 -0300,
>>>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > You can promise that well-known int32 statistic keys won't ever be higher
>>>> > than a certain value (2^18) [1] like TCP IP ports (well-known ports in
>>>> [0,
>>>> > 2^10)) but for non-standard statistics from open-source products the
>>>> key=0
>>>> > combined with string label is the way to go, otherwise collisions would
>>>> be
>>>> > inevitable and everyone would hate us for having integer keys.
>>>> >
>>>> > This is not a very weird proposal from my part because integer keys
>>>> > representing labels are common in most low-level standardized C APIs
>>>> (e.g.
>>>> > linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
>>>> > map all these keys to strings, but with them we keep the "C Data
>>>> Interface"
>>>> > low-level and portable as it should be.
>>>> >
>>>> > --
>>>> > Felipe
>>>> >
>>>> > [1] 2^16 is too small. For instance, OpenGL constants can't be enums
>>>> > because C limits enum to 2^16 and that is *not enough*.
>>>> >
>>>> > On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >> Here is an updated summary so far:
>>>> >>
>>>> >> ----
>>>> >> Use cases:
>>>> >>
>>>> >> * Optimize query plan: e.g. JOIN for DuckDB
>>>> >>
>>>> >> Out of scope:
>>>> >>
>>>> >> * Transmit statistics through not the C data interface
>>>> >>   Examples:
>>>> >>   * Transmit statistics through Apache Arrow IPC file
>>>> >>   * Transmit statistics through Apache Arrow Flight
>>>> >> * Multi-column statistics
>>>> >> * Constraints information
>>>> >> * Indexes information
>>>> >>
>>>> >> Discussing approach:
>>>> >>
>>>> >> Standardize Apache Arrow schema for statistics and transmit
>>>> >> statistics via separated API call that uses the C data
>>>> >> interface.
>>>> >>
>>>> >> This also works for per-batch statistics.
>>>> >>
>>>> >> Candidate schema:
>>>> >>
>>>> >> map<
>>>> >>   // The column index or null if the statistics refer to whole table or
>>>> >> batch.
>>>> >>   column: int32,
>>>> >>   // Statistics key is int32.
>>>> >>   // Different keys are assigned for exact value and
>>>> >>   // approximate value.
>>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>>> >> keys...>>
>>>> >> >
>>>> >>
>>>> >> Discussions:
>>>> >>
>>>> >> 1. Can we use int32 for statistic keys?
>>>> >>    Should we use utf8 (or dictionary<int32, utf8>) for
>>>> >>    statistic keys?
>>>> >> 2. Hot to support non-standard (vendor-specific)
>>>> >>    statistic keys?
>>>> >> ----
>>>> >>
>>>> >> Here is my idea:
>>>> >>
>>>> >> 1. We can use int32 for statistic keys.
>>>> >> 2. We can reserve a specific range for non-standard
>>>> >>    statistic keys. Prerequisites of this:
>>>> >>    * There is no use case to merge some statistics for
>>>> >>      the same data.
>>>> >>    * We can't merge statistics for different data.
>>>> >>
>>>> >> If the prerequisites aren't satisfied:
>>>> >>
>>>> >> 1. We should use utf8 (or dictionary<int32, utf8>) for
>>>> >>    statistic keys?
>>>> >> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>>>> >>    standard statistic keys or use prefix such as
>>>> >>    "vendor1:"/"vendor1." for non-standard statistic keys.
>>>> >>
>>>> >> Here is Felipe's idea:
>>>> >> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>>>> >>
>>>> >> 1. We can use int32 for statistic keys.
>>>> >> 2. We can use the special statistic key + a string identifier
>>>> >>    for non-standard statistic keys.
>>>> >>
>>>> >>
>>>> >> What do you think about this?
>>>> >>
>>>> >>
>>>> >> Thanks,
>>>> >> --
>>>> >> kou
>>>> >>
>>>> >> In <20...@clear-code.com>
>>>> >>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
>>>> >> 2024 18:27:27 +0900 (JST),
>>>> >>   Sutou Kouhei <ko...@clear-code.com> wrote:
>>>> >>
>>>> >> > Hi,
>>>> >> >
>>>> >> > Thanks for sharing your comments. Here is a summary so far:
>>>> >> >
>>>> >> > ----
>>>> >> >
>>>> >> > Use cases:
>>>> >> >
>>>> >> > * Optimize query plan: e.g. JOIN for DuckDB
>>>> >> >
>>>> >> > Out of scope:
>>>> >> >
>>>> >> > * Transmit statistics through not the C data interface
>>>> >> >   Examples:
>>>> >> >   * Transmit statistics through Apache Arrow IPC file
>>>> >> >   * Transmit statistics through Apache Arrow Flight
>>>> >> >
>>>> >> > Candidate approaches:
>>>> >> >
>>>> >> > 1. Pass statistics (encoded as an Apache Arrow data) via
>>>> >> >    ArrowSchema metadata
>>>> >> >    * This embeds statistics address into metadata
>>>> >> >    * It's for avoiding using Apache Arrow IPC format with
>>>> >> >      the C data interface
>>>> >> > 2. Embed statistics (encoded as an Apache Arrow data) into
>>>> >> >    ArrowSchema metadata
>>>> >> >    * This adds statistics to metadata in Apache Arrow IPC
>>>> >> >      format
>>>> >> > 3. Embed statistics (encoded as JSON) into ArrowArray
>>>> >> >    metadata
>>>> >> > 4. Standardize Apache Arrow schema for statistics and
>>>> >> >    transmit statistics via separated API call that uses the
>>>> >> >    C data interface
>>>> >> > 5. Use ADBC
>>>> >> >
>>>> >> > ----
>>>> >> >
>>>> >> > I think that 4. is the best approach in these candidates.
>>>> >> >
>>>> >> > 1. Embedding statistics address is tricky.
>>>> >> > 2. Consumers need to parse Apache Arrow IPC format data.
>>>> >> >    (The C data interface consumers may not have the
>>>> >> >    feature.)
>>>> >> > 3. This will work but 4. is more generic.
>>>> >> > 5. ADBC is too large to use only for statistics.
>>>> >> >
>>>> >> > What do you think about this?
>>>> >> >
>>>> >> >
>>>> >> > If we select 4., we need to standardize Apache Arrow schema
>>>> >> > for statistics. How about the following schema?
>>>> >> >
>>>> >> > ----
>>>> >> > Metadata:
>>>> >> >
>>>> >> > | Name                       | Value | Comments |
>>>> >> > |----------------------------|-------|--------- |
>>>> >> > | ARROW::statistics::version | 1.0.0 | (1)      |
>>>> >> >
>>>> >> > (1) This follows semantic versioning.
>>>> >> >
>>>> >> > Fields:
>>>> >> >
>>>> >> > | Name           | Type                  | Comments |
>>>> >> > |----------------|-----------------------| -------- |
>>>> >> > | column         | utf8                  | (2)      |
>>>> >> > | key            | utf8 not null         | (3)      |
>>>> >> > | value          | VALUE_SCHEMA not null |          |
>>>> >> > | is_approximate | bool not null         | (4)      |
>>>> >> >
>>>> >> > (2) If null, then the statistic applies to the entire table.
>>>> >> >     It's for "row_count".
>>>> >> > (3) We'll provide pre-defined keys such as "max", "min",
>>>> >> >     "byte_width" and "distinct_count" but users can also use
>>>> >> >     application specific keys.
>>>> >> > (4) If true, then the value is approximate or best-effort.
>>>> >> >
>>>> >> > VALUE_SCHEMA is a dense union with members:
>>>> >> >
>>>> >> > | Name    | Type    |
>>>> >> > |---------|---------|
>>>> >> > | int64   | int64   |
>>>> >> > | uint64  | uint64  |
>>>> >> > | float64 | float64 |
>>>> >> > | binary  | binary  |
>>>> >> >
>>>> >> > If a column is an int32 column, it uses int64 for
>>>> >> > "max"/"min". We don't provide all types here. Users should
>>>> >> > use a compatible type (int64 for a int32 column) instead.
>>>> >> > ----
>>>> >> >
>>>> >> >
>>>> >> > Thanks,
>>>> >> > --
>>>> >> > kou
>>>> >> >
>>>> >> >
>>>> >> > In <20...@clear-code.com>
>>>> >> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
>>>> >> 2024 11:37:08 +0900 (JST),
>>>> >> >   Sutou Kouhei <ko...@clear-code.com> wrote:
>>>> >> >
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> We're discussing how to provide statistics through the C
>>>> >> >> data interface at:
>>>> >> >> https://github.com/apache/arrow/issues/38837
>>>> >> >>
>>>> >> >> If you're interested in this feature, could you share your
>>>> >> >> comments?
>>>> >> >>
>>>> >> >>
>>>> >> >> Motivation:
>>>> >> >>
>>>> >> >> We can interchange Apache Arrow data by the C data interface
>>>> >> >> in the same process. For example, we can pass Apache Arrow
>>>> >> >> data read by Apache Arrow C++ (provider) to DuckDB
>>>> >> >> (consumer) through the C data interface.
>>>> >> >>
>>>> >> >> A provider may know Apache Arrow data statistics. For
>>>> >> >> example, a provider can know statistics when it reads Apache
>>>> >> >> Parquet data because Apache Parquet may provide statistics.
>>>> >> >>
>>>> >> >> But a consumer can't know statistics that are known by a
>>>> >> >> producer. Because there isn't a standard way to provide
>>>> >> >> statistics through the C data interface. If a consumer can
>>>> >> >> know statistics, it can process Apache Arrow data faster
>>>> >> >> based on statistics.
>>>> >> >>
>>>> >> >>
>>>> >> >> Proposal:
>>>> >> >>
>>>> >> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
>>>> >> >>
>>>> >> >> How about providing statistics as a metadata in ArrowSchema?
>>>> >> >>
>>>> >> >> We reserve "ARROW" namespace for internal Apache Arrow use:
>>>> >> >>
>>>> >> >>
>>>> >>
>>>> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
>>>> >> >>
>>>> >> >>> The ARROW pattern is a reserved namespace for internal
>>>> >> >>> Arrow use in the custom_metadata fields. For example,
>>>> >> >>> ARROW:extension:name.
>>>> >> >>
>>>> >> >> So we can use "ARROW:statistics" for the metadata key.
>>>> >> >>
>>>> >> >> We can represent statistics as a ArrowArray like ADBC does.
>>>> >> >>
>>>> >> >> Here is an example ArrowSchema that is for a record batch
>>>> >> >> that has "int32 column1" and "string column2":
>>>> >> >>
>>>> >> >> ArrowSchema {
>>>> >> >>   .format = "+siu",
>>>> >> >>   .metadata = {
>>>> >> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
>>>> >> as row count */
>>>> >> >>   },
>>>> >> >>   .children = {
>>>> >> >>     ArrowSchema {
>>>> >> >>       .name = "column1",
>>>> >> >>       .format = "i",
>>>> >> >>       .metadata = {
>>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>>> >> such as count distinct */
>>>> >> >>       },
>>>> >> >>     },
>>>> >> >>     ArrowSchema {
>>>> >> >>       .name = "column2",
>>>> >> >>       .format = "u",
>>>> >> >>       .metadata = {
>>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>>> >> such as count distinct */
>>>> >> >>       },
>>>> >> >>     },
>>>> >> >>   },
>>>> >> >> }
>>>> >> >>
>>>> >> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
>>>> >> >> => ArrowArray*' is a base 10 string of the address of the
>>>> >> >> ArrowArray. Because we can use only string for metadata
>>>> >> >> value. You can't release the statistics ArrowArray*. (Its
>>>> >> >> release is a no-op function.) It follows
>>>> >> >>
>>>> >>
>>>> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
>>>> >> >> semantics. (The base ArrowSchema owns statistics
>>>> >> >> ArrowArray*.)
>>>> >> >>
>>>> >> >>
>>>> >> >> ArrowArray* for statistics use the following schema:
>>>> >> >>
>>>> >> >> | Field Name     | Field Type                       | Comments |
>>>> >> >> |----------------|----------------------------------| -------- |
>>>> >> >> | key            | string not null                  | (1)      |
>>>> >> >> | value          | `VALUE_SCHEMA` not null          |          |
>>>> >> >> | is_approximate | bool not null                    | (2)      |
>>>> >> >>
>>>> >> >> 1. We'll provide pre-defined keys such as "max", "min",
>>>> >> >>    "byte_width" and "distinct_count" but users can also use
>>>> >> >>    application specific keys.
>>>> >> >>
>>>> >> >> 2. If true, then the value is approximate or best-effort.
>>>> >> >>
>>>> >> >> VALUE_SCHEMA is a dense union with members:
>>>> >> >>
>>>> >> >> | Field Name | Field Type                       | Comments |
>>>> >> >> |------------|----------------------------------| -------- |
>>>> >> >> | int64      | int64                            |          |
>>>> >> >> | uint64     | uint64                           |          |
>>>> >> >> | float64    | float64                          |          |
>>>> >> >> | value      | The same type of the ArrowSchema | (3)      |
>>>> >> >> |            | that is belonged to.             |          |
>>>> >> >>
>>>> >> >> 3. If the ArrowSchema's type is string, this type is also string.
>>>> >> >>
>>>> >> >>    TODO: Is "value" good name? If we refer it from the
>>>> >> >>    top-level statistics schema, we need to use
>>>> >> >>    "value.value". It's a bit strange...
>>>> >> >>
>>>> >> >>
>>>> >> >> What do you think about this proposal? Could you share your
>>>> >> >> comments?
>>>> >> >>
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> --
>>>> >> >> kou
>>>> >>
>>>>

Re: [DISCUSS] Statistics through the C data interface

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

If nobody objects using utf8 or dictionary<int32, utf8> for
statistics key, let's use dictionary<int32, utf8>. Because
dictionary<int32, utf8> will be more effective than utf8
when there are many columns.

I'll start writing a documentation for this and implementing
this for C++ next week. I'll share a PR for them when I
complete them. We can start a vote for this after we review
the PR.


Thanks,
-- 
kou

In <20...@clear-code.com>
  "Re: [DISCUSS] Statistics through the C data interface" on Fri, 12 Jul 2024 15:15:36 +0900 (JST),
  Sutou Kouhei <ko...@clear-code.com> wrote:

> Hi,
> 
>>> map<struct<int32, utf8>,
>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>
>> 
>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>> unions gracefully, this could be:
>> 
>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>> in the keys...>>
>> 
>> X is either sparse or dense.
>> 
>> A possible alternative is to use a custom struct instead of map and reduce
>> the levels of nesting:
>> 
>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
> 
> Thanks for clarifying your suggestion.
> 
> If we need utf8 for non-standard statistics, I think that
> map<utf8, ...> or map<dictionary<int32, utf8>, ...> is
> better as Antoine said. Because they are simpler than
> int32+utf8.
> 
> 
> Thanks,
> -- 
> kou
> 
> In <CA...@mail.gmail.com>
>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 11 Jul 2024 14:17:46 -0300,
>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
> 
>> On Thu, Jul 11, 2024 at 5:04 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>> 
>>> Hi,
>>>
>>> >            for non-standard statistics from open-source products the
>>> key=0
>>> > combined with string label is the way to go
>>>
>>> Where do we store the string label?
>>>
>>> I think that we're considering the following schema:
>>>
>>> >> map<
>>> >>   // The column index or null if the statistics refer to whole table or
>>> batch.
>>> >>   column: int32,
>>> >>   // Statistics key is int32.
>>> >>   // Different keys are assigned for exact value and
>>> >>   // approximate value.
>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>> keys...>>
>>> >> >
>>>
>>> Are you considering the following schema for key=0 case?
>>>
>>> map<struct<int32, utf8>,
>>>     dense_union<...needed types based on stat kinds in the keys...>>
>>>
>> 
>> Yes. That's my suggestion. And to leverage the fact that libraries handles
>> unions gracefully, this could be:
>> 
>> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
>> in the keys...>>
>> 
>> X is either sparse or dense.
>> 
>> A possible alternative is to use a custom struct instead of map and reduce
>> the levels of nesting:
>> 
>> struct<int32, utf8, dense_union<...needed types base on the keys...>>
>> 
>> --
>> Felipe
>> 
>> 
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>> In <CA...@mail.gmail.com>
>>>   "Re: [DISCUSS] Statistics through the C data interface" on Mon, 1 Jul
>>> 2024 11:58:44 -0300,
>>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > You can promise that well-known int32 statistic keys won't ever be higher
>>> > than a certain value (2^18) [1] like TCP IP ports (well-known ports in
>>> [0,
>>> > 2^10)) but for non-standard statistics from open-source products the
>>> key=0
>>> > combined with string label is the way to go, otherwise collisions would
>>> be
>>> > inevitable and everyone would hate us for having integer keys.
>>> >
>>> > This is not a very weird proposal from my part because integer keys
>>> > representing labels are common in most low-level standardized C APIs
>>> (e.g.
>>> > linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
>>> > map all these keys to strings, but with them we keep the "C Data
>>> Interface"
>>> > low-level and portable as it should be.
>>> >
>>> > --
>>> > Felipe
>>> >
>>> > [1] 2^16 is too small. For instance, OpenGL constants can't be enums
>>> > because C limits enum to 2^16 and that is *not enough*.
>>> >
>>> > On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> Here is an updated summary so far:
>>> >>
>>> >> ----
>>> >> Use cases:
>>> >>
>>> >> * Optimize query plan: e.g. JOIN for DuckDB
>>> >>
>>> >> Out of scope:
>>> >>
>>> >> * Transmit statistics through not the C data interface
>>> >>   Examples:
>>> >>   * Transmit statistics through Apache Arrow IPC file
>>> >>   * Transmit statistics through Apache Arrow Flight
>>> >> * Multi-column statistics
>>> >> * Constraints information
>>> >> * Indexes information
>>> >>
>>> >> Discussing approach:
>>> >>
>>> >> Standardize Apache Arrow schema for statistics and transmit
>>> >> statistics via separated API call that uses the C data
>>> >> interface.
>>> >>
>>> >> This also works for per-batch statistics.
>>> >>
>>> >> Candidate schema:
>>> >>
>>> >> map<
>>> >>   // The column index or null if the statistics refer to whole table or
>>> >> batch.
>>> >>   column: int32,
>>> >>   // Statistics key is int32.
>>> >>   // Different keys are assigned for exact value and
>>> >>   // approximate value.
>>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>>> >> keys...>>
>>> >> >
>>> >>
>>> >> Discussions:
>>> >>
>>> >> 1. Can we use int32 for statistic keys?
>>> >>    Should we use utf8 (or dictionary<int32, utf8>) for
>>> >>    statistic keys?
>>> >> 2. Hot to support non-standard (vendor-specific)
>>> >>    statistic keys?
>>> >> ----
>>> >>
>>> >> Here is my idea:
>>> >>
>>> >> 1. We can use int32 for statistic keys.
>>> >> 2. We can reserve a specific range for non-standard
>>> >>    statistic keys. Prerequisites of this:
>>> >>    * There is no use case to merge some statistics for
>>> >>      the same data.
>>> >>    * We can't merge statistics for different data.
>>> >>
>>> >> If the prerequisites aren't satisfied:
>>> >>
>>> >> 1. We should use utf8 (or dictionary<int32, utf8>) for
>>> >>    statistic keys?
>>> >> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>>> >>    standard statistic keys or use prefix such as
>>> >>    "vendor1:"/"vendor1." for non-standard statistic keys.
>>> >>
>>> >> Here is Felipe's idea:
>>> >> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>>> >>
>>> >> 1. We can use int32 for statistic keys.
>>> >> 2. We can use the special statistic key + a string identifier
>>> >>    for non-standard statistic keys.
>>> >>
>>> >>
>>> >> What do you think about this?
>>> >>
>>> >>
>>> >> Thanks,
>>> >> --
>>> >> kou
>>> >>
>>> >> In <20...@clear-code.com>
>>> >>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
>>> >> 2024 18:27:27 +0900 (JST),
>>> >>   Sutou Kouhei <ko...@clear-code.com> wrote:
>>> >>
>>> >> > Hi,
>>> >> >
>>> >> > Thanks for sharing your comments. Here is a summary so far:
>>> >> >
>>> >> > ----
>>> >> >
>>> >> > Use cases:
>>> >> >
>>> >> > * Optimize query plan: e.g. JOIN for DuckDB
>>> >> >
>>> >> > Out of scope:
>>> >> >
>>> >> > * Transmit statistics through not the C data interface
>>> >> >   Examples:
>>> >> >   * Transmit statistics through Apache Arrow IPC file
>>> >> >   * Transmit statistics through Apache Arrow Flight
>>> >> >
>>> >> > Candidate approaches:
>>> >> >
>>> >> > 1. Pass statistics (encoded as an Apache Arrow data) via
>>> >> >    ArrowSchema metadata
>>> >> >    * This embeds statistics address into metadata
>>> >> >    * It's for avoiding using Apache Arrow IPC format with
>>> >> >      the C data interface
>>> >> > 2. Embed statistics (encoded as an Apache Arrow data) into
>>> >> >    ArrowSchema metadata
>>> >> >    * This adds statistics to metadata in Apache Arrow IPC
>>> >> >      format
>>> >> > 3. Embed statistics (encoded as JSON) into ArrowArray
>>> >> >    metadata
>>> >> > 4. Standardize Apache Arrow schema for statistics and
>>> >> >    transmit statistics via separated API call that uses the
>>> >> >    C data interface
>>> >> > 5. Use ADBC
>>> >> >
>>> >> > ----
>>> >> >
>>> >> > I think that 4. is the best approach in these candidates.
>>> >> >
>>> >> > 1. Embedding statistics address is tricky.
>>> >> > 2. Consumers need to parse Apache Arrow IPC format data.
>>> >> >    (The C data interface consumers may not have the
>>> >> >    feature.)
>>> >> > 3. This will work but 4. is more generic.
>>> >> > 5. ADBC is too large to use only for statistics.
>>> >> >
>>> >> > What do you think about this?
>>> >> >
>>> >> >
>>> >> > If we select 4., we need to standardize Apache Arrow schema
>>> >> > for statistics. How about the following schema?
>>> >> >
>>> >> > ----
>>> >> > Metadata:
>>> >> >
>>> >> > | Name                       | Value | Comments |
>>> >> > |----------------------------|-------|--------- |
>>> >> > | ARROW::statistics::version | 1.0.0 | (1)      |
>>> >> >
>>> >> > (1) This follows semantic versioning.
>>> >> >
>>> >> > Fields:
>>> >> >
>>> >> > | Name           | Type                  | Comments |
>>> >> > |----------------|-----------------------| -------- |
>>> >> > | column         | utf8                  | (2)      |
>>> >> > | key            | utf8 not null         | (3)      |
>>> >> > | value          | VALUE_SCHEMA not null |          |
>>> >> > | is_approximate | bool not null         | (4)      |
>>> >> >
>>> >> > (2) If null, then the statistic applies to the entire table.
>>> >> >     It's for "row_count".
>>> >> > (3) We'll provide pre-defined keys such as "max", "min",
>>> >> >     "byte_width" and "distinct_count" but users can also use
>>> >> >     application specific keys.
>>> >> > (4) If true, then the value is approximate or best-effort.
>>> >> >
>>> >> > VALUE_SCHEMA is a dense union with members:
>>> >> >
>>> >> > | Name    | Type    |
>>> >> > |---------|---------|
>>> >> > | int64   | int64   |
>>> >> > | uint64  | uint64  |
>>> >> > | float64 | float64 |
>>> >> > | binary  | binary  |
>>> >> >
>>> >> > If a column is an int32 column, it uses int64 for
>>> >> > "max"/"min". We don't provide all types here. Users should
>>> >> > use a compatible type (int64 for a int32 column) instead.
>>> >> > ----
>>> >> >
>>> >> >
>>> >> > Thanks,
>>> >> > --
>>> >> > kou
>>> >> >
>>> >> >
>>> >> > In <20...@clear-code.com>
>>> >> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
>>> >> 2024 11:37:08 +0900 (JST),
>>> >> >   Sutou Kouhei <ko...@clear-code.com> wrote:
>>> >> >
>>> >> >> Hi,
>>> >> >>
>>> >> >> We're discussing how to provide statistics through the C
>>> >> >> data interface at:
>>> >> >> https://github.com/apache/arrow/issues/38837
>>> >> >>
>>> >> >> If you're interested in this feature, could you share your
>>> >> >> comments?
>>> >> >>
>>> >> >>
>>> >> >> Motivation:
>>> >> >>
>>> >> >> We can interchange Apache Arrow data by the C data interface
>>> >> >> in the same process. For example, we can pass Apache Arrow
>>> >> >> data read by Apache Arrow C++ (provider) to DuckDB
>>> >> >> (consumer) through the C data interface.
>>> >> >>
>>> >> >> A provider may know Apache Arrow data statistics. For
>>> >> >> example, a provider can know statistics when it reads Apache
>>> >> >> Parquet data because Apache Parquet may provide statistics.
>>> >> >>
>>> >> >> But a consumer can't know statistics that are known by a
>>> >> >> producer. Because there isn't a standard way to provide
>>> >> >> statistics through the C data interface. If a consumer can
>>> >> >> know statistics, it can process Apache Arrow data faster
>>> >> >> based on statistics.
>>> >> >>
>>> >> >>
>>> >> >> Proposal:
>>> >> >>
>>> >> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
>>> >> >>
>>> >> >> How about providing statistics as a metadata in ArrowSchema?
>>> >> >>
>>> >> >> We reserve "ARROW" namespace for internal Apache Arrow use:
>>> >> >>
>>> >> >>
>>> >>
>>> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
>>> >> >>
>>> >> >>> The ARROW pattern is a reserved namespace for internal
>>> >> >>> Arrow use in the custom_metadata fields. For example,
>>> >> >>> ARROW:extension:name.
>>> >> >>
>>> >> >> So we can use "ARROW:statistics" for the metadata key.
>>> >> >>
>>> >> >> We can represent statistics as a ArrowArray like ADBC does.
>>> >> >>
>>> >> >> Here is an example ArrowSchema that is for a record batch
>>> >> >> that has "int32 column1" and "string column2":
>>> >> >>
>>> >> >> ArrowSchema {
>>> >> >>   .format = "+siu",
>>> >> >>   .metadata = {
>>> >> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
>>> >> as row count */
>>> >> >>   },
>>> >> >>   .children = {
>>> >> >>     ArrowSchema {
>>> >> >>       .name = "column1",
>>> >> >>       .format = "i",
>>> >> >>       .metadata = {
>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>> >> such as count distinct */
>>> >> >>       },
>>> >> >>     },
>>> >> >>     ArrowSchema {
>>> >> >>       .name = "column2",
>>> >> >>       .format = "u",
>>> >> >>       .metadata = {
>>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>>> >> such as count distinct */
>>> >> >>       },
>>> >> >>     },
>>> >> >>   },
>>> >> >> }
>>> >> >>
>>> >> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
>>> >> >> => ArrowArray*' is a base 10 string of the address of the
>>> >> >> ArrowArray. Because we can use only string for metadata
>>> >> >> value. You can't release the statistics ArrowArray*. (Its
>>> >> >> release is a no-op function.) It follows
>>> >> >>
>>> >>
>>> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
>>> >> >> semantics. (The base ArrowSchema owns statistics
>>> >> >> ArrowArray*.)
>>> >> >>
>>> >> >>
>>> >> >> ArrowArray* for statistics use the following schema:
>>> >> >>
>>> >> >> | Field Name     | Field Type                       | Comments |
>>> >> >> |----------------|----------------------------------| -------- |
>>> >> >> | key            | string not null                  | (1)      |
>>> >> >> | value          | `VALUE_SCHEMA` not null          |          |
>>> >> >> | is_approximate | bool not null                    | (2)      |
>>> >> >>
>>> >> >> 1. We'll provide pre-defined keys such as "max", "min",
>>> >> >>    "byte_width" and "distinct_count" but users can also use
>>> >> >>    application specific keys.
>>> >> >>
>>> >> >> 2. If true, then the value is approximate or best-effort.
>>> >> >>
>>> >> >> VALUE_SCHEMA is a dense union with members:
>>> >> >>
>>> >> >> | Field Name | Field Type                       | Comments |
>>> >> >> |------------|----------------------------------| -------- |
>>> >> >> | int64      | int64                            |          |
>>> >> >> | uint64     | uint64                           |          |
>>> >> >> | float64    | float64                          |          |
>>> >> >> | value      | The same type of the ArrowSchema | (3)      |
>>> >> >> |            | that is belonged to.             |          |
>>> >> >>
>>> >> >> 3. If the ArrowSchema's type is string, this type is also string.
>>> >> >>
>>> >> >>    TODO: Is "value" good name? If we refer it from the
>>> >> >>    top-level statistics schema, we need to use
>>> >> >>    "value.value". It's a bit strange...
>>> >> >>
>>> >> >>
>>> >> >> What do you think about this proposal? Could you share your
>>> >> >> comments?
>>> >> >>
>>> >> >>
>>> >> >> Thanks,
>>> >> >> --
>>> >> >> kou
>>> >>
>>>

Re: [DISCUSS] Statistics through the C data interface

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

>> map<struct<int32, utf8>,
>>     dense_union<...needed types based on stat kinds in the keys...>>
>>
> 
> Yes. That's my suggestion. And to leverage the fact that libraries handles
> unions gracefully, this could be:
> 
> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
> in the keys...>>
> 
> X is either sparse or dense.
> 
> A possible alternative is to use a custom struct instead of map and reduce
> the levels of nesting:
> 
> struct<int32, utf8, dense_union<...needed types base on the keys...>>

Thanks for clarifying your suggestion.

If we need utf8 for non-standard statistics, I think that
map<utf8, ...> or map<dictionary<int32, utf8>, ...> is
better as Antoine said. Because they are simpler than
int32+utf8.


Thanks,
-- 
kou

In <CA...@mail.gmail.com>
  "Re: [DISCUSS] Statistics through the C data interface" on Thu, 11 Jul 2024 14:17:46 -0300,
  Felipe Oliveira Carvalho <fe...@gmail.com> wrote:

> On Thu, Jul 11, 2024 at 5:04 AM Sutou Kouhei <ko...@clear-code.com> wrote:
> 
>> Hi,
>>
>> >            for non-standard statistics from open-source products the
>> key=0
>> > combined with string label is the way to go
>>
>> Where do we store the string label?
>>
>> I think that we're considering the following schema:
>>
>> >> map<
>> >>   // The column index or null if the statistics refer to whole table or
>> batch.
>> >>   column: int32,
>> >>   // Statistics key is int32.
>> >>   // Different keys are assigned for exact value and
>> >>   // approximate value.
>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>> keys...>>
>> >> >
>>
>> Are you considering the following schema for key=0 case?
>>
>> map<struct<int32, utf8>,
>>     dense_union<...needed types based on stat kinds in the keys...>>
>>
> 
> Yes. That's my suggestion. And to leverage the fact that libraries handles
> unions gracefully, this could be:
> 
> map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
> in the keys...>>
> 
> X is either sparse or dense.
> 
> A possible alternative is to use a custom struct instead of map and reduce
> the levels of nesting:
> 
> struct<int32, utf8, dense_union<...needed types base on the keys...>>
> 
> --
> Felipe
> 
> 
>>
>> Thanks,
>> --
>> kou
>>
>> In <CA...@mail.gmail.com>
>>   "Re: [DISCUSS] Statistics through the C data interface" on Mon, 1 Jul
>> 2024 11:58:44 -0300,
>>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > You can promise that well-known int32 statistic keys won't ever be higher
>> > than a certain value (2^18) [1] like TCP IP ports (well-known ports in
>> [0,
>> > 2^10)) but for non-standard statistics from open-source products the
>> key=0
>> > combined with string label is the way to go, otherwise collisions would
>> be
>> > inevitable and everyone would hate us for having integer keys.
>> >
>> > This is not a very weird proposal from my part because integer keys
>> > representing labels are common in most low-level standardized C APIs
>> (e.g.
>> > linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
>> > map all these keys to strings, but with them we keep the "C Data
>> Interface"
>> > low-level and portable as it should be.
>> >
>> > --
>> > Felipe
>> >
>> > [1] 2^16 is too small. For instance, OpenGL constants can't be enums
>> > because C limits enum to 2^16 and that is *not enough*.
>> >
>> > On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> Here is an updated summary so far:
>> >>
>> >> ----
>> >> Use cases:
>> >>
>> >> * Optimize query plan: e.g. JOIN for DuckDB
>> >>
>> >> Out of scope:
>> >>
>> >> * Transmit statistics through not the C data interface
>> >>   Examples:
>> >>   * Transmit statistics through Apache Arrow IPC file
>> >>   * Transmit statistics through Apache Arrow Flight
>> >> * Multi-column statistics
>> >> * Constraints information
>> >> * Indexes information
>> >>
>> >> Discussing approach:
>> >>
>> >> Standardize Apache Arrow schema for statistics and transmit
>> >> statistics via separated API call that uses the C data
>> >> interface.
>> >>
>> >> This also works for per-batch statistics.
>> >>
>> >> Candidate schema:
>> >>
>> >> map<
>> >>   // The column index or null if the statistics refer to whole table or
>> >> batch.
>> >>   column: int32,
>> >>   // Statistics key is int32.
>> >>   // Different keys are assigned for exact value and
>> >>   // approximate value.
>> >>   map<int32, dense_union<...needed types based on stat kinds in the
>> >> keys...>>
>> >> >
>> >>
>> >> Discussions:
>> >>
>> >> 1. Can we use int32 for statistic keys?
>> >>    Should we use utf8 (or dictionary<int32, utf8>) for
>> >>    statistic keys?
>> >> 2. Hot to support non-standard (vendor-specific)
>> >>    statistic keys?
>> >> ----
>> >>
>> >> Here is my idea:
>> >>
>> >> 1. We can use int32 for statistic keys.
>> >> 2. We can reserve a specific range for non-standard
>> >>    statistic keys. Prerequisites of this:
>> >>    * There is no use case to merge some statistics for
>> >>      the same data.
>> >>    * We can't merge statistics for different data.
>> >>
>> >> If the prerequisites aren't satisfied:
>> >>
>> >> 1. We should use utf8 (or dictionary<int32, utf8>) for
>> >>    statistic keys?
>> >> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>> >>    standard statistic keys or use prefix such as
>> >>    "vendor1:"/"vendor1." for non-standard statistic keys.
>> >>
>> >> Here is Felipe's idea:
>> >> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>> >>
>> >> 1. We can use int32 for statistic keys.
>> >> 2. We can use the special statistic key + a string identifier
>> >>    for non-standard statistic keys.
>> >>
>> >>
>> >> What do you think about this?
>> >>
>> >>
>> >> Thanks,
>> >> --
>> >> kou
>> >>
>> >> In <20...@clear-code.com>
>> >>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
>> >> 2024 18:27:27 +0900 (JST),
>> >>   Sutou Kouhei <ko...@clear-code.com> wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > Thanks for sharing your comments. Here is a summary so far:
>> >> >
>> >> > ----
>> >> >
>> >> > Use cases:
>> >> >
>> >> > * Optimize query plan: e.g. JOIN for DuckDB
>> >> >
>> >> > Out of scope:
>> >> >
>> >> > * Transmit statistics through not the C data interface
>> >> >   Examples:
>> >> >   * Transmit statistics through Apache Arrow IPC file
>> >> >   * Transmit statistics through Apache Arrow Flight
>> >> >
>> >> > Candidate approaches:
>> >> >
>> >> > 1. Pass statistics (encoded as an Apache Arrow data) via
>> >> >    ArrowSchema metadata
>> >> >    * This embeds statistics address into metadata
>> >> >    * It's for avoiding using Apache Arrow IPC format with
>> >> >      the C data interface
>> >> > 2. Embed statistics (encoded as an Apache Arrow data) into
>> >> >    ArrowSchema metadata
>> >> >    * This adds statistics to metadata in Apache Arrow IPC
>> >> >      format
>> >> > 3. Embed statistics (encoded as JSON) into ArrowArray
>> >> >    metadata
>> >> > 4. Standardize Apache Arrow schema for statistics and
>> >> >    transmit statistics via separated API call that uses the
>> >> >    C data interface
>> >> > 5. Use ADBC
>> >> >
>> >> > ----
>> >> >
>> >> > I think that 4. is the best approach in these candidates.
>> >> >
>> >> > 1. Embedding statistics address is tricky.
>> >> > 2. Consumers need to parse Apache Arrow IPC format data.
>> >> >    (The C data interface consumers may not have the
>> >> >    feature.)
>> >> > 3. This will work but 4. is more generic.
>> >> > 5. ADBC is too large to use only for statistics.
>> >> >
>> >> > What do you think about this?
>> >> >
>> >> >
>> >> > If we select 4., we need to standardize Apache Arrow schema
>> >> > for statistics. How about the following schema?
>> >> >
>> >> > ----
>> >> > Metadata:
>> >> >
>> >> > | Name                       | Value | Comments |
>> >> > |----------------------------|-------|--------- |
>> >> > | ARROW::statistics::version | 1.0.0 | (1)      |
>> >> >
>> >> > (1) This follows semantic versioning.
>> >> >
>> >> > Fields:
>> >> >
>> >> > | Name           | Type                  | Comments |
>> >> > |----------------|-----------------------| -------- |
>> >> > | column         | utf8                  | (2)      |
>> >> > | key            | utf8 not null         | (3)      |
>> >> > | value          | VALUE_SCHEMA not null |          |
>> >> > | is_approximate | bool not null         | (4)      |
>> >> >
>> >> > (2) If null, then the statistic applies to the entire table.
>> >> >     It's for "row_count".
>> >> > (3) We'll provide pre-defined keys such as "max", "min",
>> >> >     "byte_width" and "distinct_count" but users can also use
>> >> >     application specific keys.
>> >> > (4) If true, then the value is approximate or best-effort.
>> >> >
>> >> > VALUE_SCHEMA is a dense union with members:
>> >> >
>> >> > | Name    | Type    |
>> >> > |---------|---------|
>> >> > | int64   | int64   |
>> >> > | uint64  | uint64  |
>> >> > | float64 | float64 |
>> >> > | binary  | binary  |
>> >> >
>> >> > If a column is an int32 column, it uses int64 for
>> >> > "max"/"min". We don't provide all types here. Users should
>> >> > use a compatible type (int64 for a int32 column) instead.
>> >> > ----
>> >> >
>> >> >
>> >> > Thanks,
>> >> > --
>> >> > kou
>> >> >
>> >> >
>> >> > In <20...@clear-code.com>
>> >> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
>> >> 2024 11:37:08 +0900 (JST),
>> >> >   Sutou Kouhei <ko...@clear-code.com> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> We're discussing how to provide statistics through the C
>> >> >> data interface at:
>> >> >> https://github.com/apache/arrow/issues/38837
>> >> >>
>> >> >> If you're interested in this feature, could you share your
>> >> >> comments?
>> >> >>
>> >> >>
>> >> >> Motivation:
>> >> >>
>> >> >> We can interchange Apache Arrow data by the C data interface
>> >> >> in the same process. For example, we can pass Apache Arrow
>> >> >> data read by Apache Arrow C++ (provider) to DuckDB
>> >> >> (consumer) through the C data interface.
>> >> >>
>> >> >> A provider may know Apache Arrow data statistics. For
>> >> >> example, a provider can know statistics when it reads Apache
>> >> >> Parquet data because Apache Parquet may provide statistics.
>> >> >>
>> >> >> But a consumer can't know statistics that are known by a
>> >> >> producer. Because there isn't a standard way to provide
>> >> >> statistics through the C data interface. If a consumer can
>> >> >> know statistics, it can process Apache Arrow data faster
>> >> >> based on statistics.
>> >> >>
>> >> >>
>> >> >> Proposal:
>> >> >>
>> >> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
>> >> >>
>> >> >> How about providing statistics as a metadata in ArrowSchema?
>> >> >>
>> >> >> We reserve "ARROW" namespace for internal Apache Arrow use:
>> >> >>
>> >> >>
>> >>
>> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
>> >> >>
>> >> >>> The ARROW pattern is a reserved namespace for internal
>> >> >>> Arrow use in the custom_metadata fields. For example,
>> >> >>> ARROW:extension:name.
>> >> >>
>> >> >> So we can use "ARROW:statistics" for the metadata key.
>> >> >>
>> >> >> We can represent statistics as a ArrowArray like ADBC does.
>> >> >>
>> >> >> Here is an example ArrowSchema that is for a record batch
>> >> >> that has "int32 column1" and "string column2":
>> >> >>
>> >> >> ArrowSchema {
>> >> >>   .format = "+siu",
>> >> >>   .metadata = {
>> >> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
>> >> as row count */
>> >> >>   },
>> >> >>   .children = {
>> >> >>     ArrowSchema {
>> >> >>       .name = "column1",
>> >> >>       .format = "i",
>> >> >>       .metadata = {
>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>> >> such as count distinct */
>> >> >>       },
>> >> >>     },
>> >> >>     ArrowSchema {
>> >> >>       .name = "column2",
>> >> >>       .format = "u",
>> >> >>       .metadata = {
>> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>> >> such as count distinct */
>> >> >>       },
>> >> >>     },
>> >> >>   },
>> >> >> }
>> >> >>
>> >> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
>> >> >> => ArrowArray*' is a base 10 string of the address of the
>> >> >> ArrowArray. Because we can use only string for metadata
>> >> >> value. You can't release the statistics ArrowArray*. (Its
>> >> >> release is a no-op function.) It follows
>> >> >>
>> >>
>> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
>> >> >> semantics. (The base ArrowSchema owns statistics
>> >> >> ArrowArray*.)
>> >> >>
>> >> >>
>> >> >> ArrowArray* for statistics use the following schema:
>> >> >>
>> >> >> | Field Name     | Field Type                       | Comments |
>> >> >> |----------------|----------------------------------| -------- |
>> >> >> | key            | string not null                  | (1)      |
>> >> >> | value          | `VALUE_SCHEMA` not null          |          |
>> >> >> | is_approximate | bool not null                    | (2)      |
>> >> >>
>> >> >> 1. We'll provide pre-defined keys such as "max", "min",
>> >> >>    "byte_width" and "distinct_count" but users can also use
>> >> >>    application specific keys.
>> >> >>
>> >> >> 2. If true, then the value is approximate or best-effort.
>> >> >>
>> >> >> VALUE_SCHEMA is a dense union with members:
>> >> >>
>> >> >> | Field Name | Field Type                       | Comments |
>> >> >> |------------|----------------------------------| -------- |
>> >> >> | int64      | int64                            |          |
>> >> >> | uint64     | uint64                           |          |
>> >> >> | float64    | float64                          |          |
>> >> >> | value      | The same type of the ArrowSchema | (3)      |
>> >> >> |            | that is belonged to.             |          |
>> >> >>
>> >> >> 3. If the ArrowSchema's type is string, this type is also string.
>> >> >>
>> >> >>    TODO: Is "value" good name? If we refer it from the
>> >> >>    top-level statistics schema, we need to use
>> >> >>    "value.value". It's a bit strange...
>> >> >>
>> >> >>
>> >> >> What do you think about this proposal? Could you share your
>> >> >> comments?
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> --
>> >> >> kou
>> >>
>>

Re: [DISCUSS] Statistics through the C data interface

Posted by Felipe Oliveira Carvalho <fe...@gmail.com>.
On Thu, Jul 11, 2024 at 5:04 AM Sutou Kouhei <ko...@clear-code.com> wrote:

> Hi,
>
> >            for non-standard statistics from open-source products the
> key=0
> > combined with string label is the way to go
>
> Where do we store the string label?
>
> I think that we're considering the following schema:
>
> >> map<
> >>   // The column index or null if the statistics refer to whole table or
> batch.
> >>   column: int32,
> >>   // Statistics key is int32.
> >>   // Different keys are assigned for exact value and
> >>   // approximate value.
> >>   map<int32, dense_union<...needed types based on stat kinds in the
> keys...>>
> >> >
>
> Are you considering the following schema for key=0 case?
>
> map<struct<int32, utf8>,
>     dense_union<...needed types based on stat kinds in the keys...>>
>

Yes. That's my suggestion. And to leverage the fact that libraries handles
unions gracefully, this could be:

map<X_union<int32, utf8>, dense_union<...needed types based on stat kinds
in the keys...>>

X is either sparse or dense.

A possible alternative is to use a custom struct instead of map and reduce
the levels of nesting:

struct<int32, utf8, dense_union<...needed types base on the keys...>>

--
Felipe


>
> Thanks,
> --
> kou
>
> In <CA...@mail.gmail.com>
>   "Re: [DISCUSS] Statistics through the C data interface" on Mon, 1 Jul
> 2024 11:58:44 -0300,
>   Felipe Oliveira Carvalho <fe...@gmail.com> wrote:
>
> > Hi,
> >
> > You can promise that well-known int32 statistic keys won't ever be higher
> > than a certain value (2^18) [1] like TCP IP ports (well-known ports in
> [0,
> > 2^10)) but for non-standard statistics from open-source products the
> key=0
> > combined with string label is the way to go, otherwise collisions would
> be
> > inevitable and everyone would hate us for having integer keys.
> >
> > This is not a very weird proposal from my part because integer keys
> > representing labels are common in most low-level standardized C APIs
> (e.g.
> > linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
> > map all these keys to strings, but with them we keep the "C Data
> Interface"
> > low-level and portable as it should be.
> >
> > --
> > Felipe
> >
> > [1] 2^16 is too small. For instance, OpenGL constants can't be enums
> > because C limits enum to 2^16 and that is *not enough*.
> >
> > On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
> >
> >> Hi,
> >>
> >> Here is an updated summary so far:
> >>
> >> ----
> >> Use cases:
> >>
> >> * Optimize query plan: e.g. JOIN for DuckDB
> >>
> >> Out of scope:
> >>
> >> * Transmit statistics through not the C data interface
> >>   Examples:
> >>   * Transmit statistics through Apache Arrow IPC file
> >>   * Transmit statistics through Apache Arrow Flight
> >> * Multi-column statistics
> >> * Constraints information
> >> * Indexes information
> >>
> >> Discussing approach:
> >>
> >> Standardize Apache Arrow schema for statistics and transmit
> >> statistics via separated API call that uses the C data
> >> interface.
> >>
> >> This also works for per-batch statistics.
> >>
> >> Candidate schema:
> >>
> >> map<
> >>   // The column index or null if the statistics refer to whole table or
> >> batch.
> >>   column: int32,
> >>   // Statistics key is int32.
> >>   // Different keys are assigned for exact value and
> >>   // approximate value.
> >>   map<int32, dense_union<...needed types based on stat kinds in the
> >> keys...>>
> >> >
> >>
> >> Discussions:
> >>
> >> 1. Can we use int32 for statistic keys?
> >>    Should we use utf8 (or dictionary<int32, utf8>) for
> >>    statistic keys?
> >> 2. Hot to support non-standard (vendor-specific)
> >>    statistic keys?
> >> ----
> >>
> >> Here is my idea:
> >>
> >> 1. We can use int32 for statistic keys.
> >> 2. We can reserve a specific range for non-standard
> >>    statistic keys. Prerequisites of this:
> >>    * There is no use case to merge some statistics for
> >>      the same data.
> >>    * We can't merge statistics for different data.
> >>
> >> If the prerequisites aren't satisfied:
> >>
> >> 1. We should use utf8 (or dictionary<int32, utf8>) for
> >>    statistic keys?
> >> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
> >>    standard statistic keys or use prefix such as
> >>    "vendor1:"/"vendor1." for non-standard statistic keys.
> >>
> >> Here is Felipe's idea:
> >> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
> >>
> >> 1. We can use int32 for statistic keys.
> >> 2. We can use the special statistic key + a string identifier
> >>    for non-standard statistic keys.
> >>
> >>
> >> What do you think about this?
> >>
> >>
> >> Thanks,
> >> --
> >> kou
> >>
> >> In <20...@clear-code.com>
> >>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
> >> 2024 18:27:27 +0900 (JST),
> >>   Sutou Kouhei <ko...@clear-code.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > Thanks for sharing your comments. Here is a summary so far:
> >> >
> >> > ----
> >> >
> >> > Use cases:
> >> >
> >> > * Optimize query plan: e.g. JOIN for DuckDB
> >> >
> >> > Out of scope:
> >> >
> >> > * Transmit statistics through not the C data interface
> >> >   Examples:
> >> >   * Transmit statistics through Apache Arrow IPC file
> >> >   * Transmit statistics through Apache Arrow Flight
> >> >
> >> > Candidate approaches:
> >> >
> >> > 1. Pass statistics (encoded as an Apache Arrow data) via
> >> >    ArrowSchema metadata
> >> >    * This embeds statistics address into metadata
> >> >    * It's for avoiding using Apache Arrow IPC format with
> >> >      the C data interface
> >> > 2. Embed statistics (encoded as an Apache Arrow data) into
> >> >    ArrowSchema metadata
> >> >    * This adds statistics to metadata in Apache Arrow IPC
> >> >      format
> >> > 3. Embed statistics (encoded as JSON) into ArrowArray
> >> >    metadata
> >> > 4. Standardize Apache Arrow schema for statistics and
> >> >    transmit statistics via separated API call that uses the
> >> >    C data interface
> >> > 5. Use ADBC
> >> >
> >> > ----
> >> >
> >> > I think that 4. is the best approach in these candidates.
> >> >
> >> > 1. Embedding statistics address is tricky.
> >> > 2. Consumers need to parse Apache Arrow IPC format data.
> >> >    (The C data interface consumers may not have the
> >> >    feature.)
> >> > 3. This will work but 4. is more generic.
> >> > 5. ADBC is too large to use only for statistics.
> >> >
> >> > What do you think about this?
> >> >
> >> >
> >> > If we select 4., we need to standardize Apache Arrow schema
> >> > for statistics. How about the following schema?
> >> >
> >> > ----
> >> > Metadata:
> >> >
> >> > | Name                       | Value | Comments |
> >> > |----------------------------|-------|--------- |
> >> > | ARROW::statistics::version | 1.0.0 | (1)      |
> >> >
> >> > (1) This follows semantic versioning.
> >> >
> >> > Fields:
> >> >
> >> > | Name           | Type                  | Comments |
> >> > |----------------|-----------------------| -------- |
> >> > | column         | utf8                  | (2)      |
> >> > | key            | utf8 not null         | (3)      |
> >> > | value          | VALUE_SCHEMA not null |          |
> >> > | is_approximate | bool not null         | (4)      |
> >> >
> >> > (2) If null, then the statistic applies to the entire table.
> >> >     It's for "row_count".
> >> > (3) We'll provide pre-defined keys such as "max", "min",
> >> >     "byte_width" and "distinct_count" but users can also use
> >> >     application specific keys.
> >> > (4) If true, then the value is approximate or best-effort.
> >> >
> >> > VALUE_SCHEMA is a dense union with members:
> >> >
> >> > | Name    | Type    |
> >> > |---------|---------|
> >> > | int64   | int64   |
> >> > | uint64  | uint64  |
> >> > | float64 | float64 |
> >> > | binary  | binary  |
> >> >
> >> > If a column is an int32 column, it uses int64 for
> >> > "max"/"min". We don't provide all types here. Users should
> >> > use a compatible type (int64 for a int32 column) instead.
> >> > ----
> >> >
> >> >
> >> > Thanks,
> >> > --
> >> > kou
> >> >
> >> >
> >> > In <20...@clear-code.com>
> >> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
> >> 2024 11:37:08 +0900 (JST),
> >> >   Sutou Kouhei <ko...@clear-code.com> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> We're discussing how to provide statistics through the C
> >> >> data interface at:
> >> >> https://github.com/apache/arrow/issues/38837
> >> >>
> >> >> If you're interested in this feature, could you share your
> >> >> comments?
> >> >>
> >> >>
> >> >> Motivation:
> >> >>
> >> >> We can interchange Apache Arrow data by the C data interface
> >> >> in the same process. For example, we can pass Apache Arrow
> >> >> data read by Apache Arrow C++ (provider) to DuckDB
> >> >> (consumer) through the C data interface.
> >> >>
> >> >> A provider may know Apache Arrow data statistics. For
> >> >> example, a provider can know statistics when it reads Apache
> >> >> Parquet data because Apache Parquet may provide statistics.
> >> >>
> >> >> But a consumer can't know statistics that are known by a
> >> >> producer. Because there isn't a standard way to provide
> >> >> statistics through the C data interface. If a consumer can
> >> >> know statistics, it can process Apache Arrow data faster
> >> >> based on statistics.
> >> >>
> >> >>
> >> >> Proposal:
> >> >>
> >> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
> >> >>
> >> >> How about providing statistics as a metadata in ArrowSchema?
> >> >>
> >> >> We reserve "ARROW" namespace for internal Apache Arrow use:
> >> >>
> >> >>
> >>
> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
> >> >>
> >> >>> The ARROW pattern is a reserved namespace for internal
> >> >>> Arrow use in the custom_metadata fields. For example,
> >> >>> ARROW:extension:name.
> >> >>
> >> >> So we can use "ARROW:statistics" for the metadata key.
> >> >>
> >> >> We can represent statistics as a ArrowArray like ADBC does.
> >> >>
> >> >> Here is an example ArrowSchema that is for a record batch
> >> >> that has "int32 column1" and "string column2":
> >> >>
> >> >> ArrowSchema {
> >> >>   .format = "+siu",
> >> >>   .metadata = {
> >> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
> >> as row count */
> >> >>   },
> >> >>   .children = {
> >> >>     ArrowSchema {
> >> >>       .name = "column1",
> >> >>       .format = "i",
> >> >>       .metadata = {
> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
> >> such as count distinct */
> >> >>       },
> >> >>     },
> >> >>     ArrowSchema {
> >> >>       .name = "column2",
> >> >>       .format = "u",
> >> >>       .metadata = {
> >> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
> >> such as count distinct */
> >> >>       },
> >> >>     },
> >> >>   },
> >> >> }
> >> >>
> >> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
> >> >> => ArrowArray*' is a base 10 string of the address of the
> >> >> ArrowArray. Because we can use only string for metadata
> >> >> value. You can't release the statistics ArrowArray*. (Its
> >> >> release is a no-op function.) It follows
> >> >>
> >>
> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
> >> >> semantics. (The base ArrowSchema owns statistics
> >> >> ArrowArray*.)
> >> >>
> >> >>
> >> >> ArrowArray* for statistics use the following schema:
> >> >>
> >> >> | Field Name     | Field Type                       | Comments |
> >> >> |----------------|----------------------------------| -------- |
> >> >> | key            | string not null                  | (1)      |
> >> >> | value          | `VALUE_SCHEMA` not null          |          |
> >> >> | is_approximate | bool not null                    | (2)      |
> >> >>
> >> >> 1. We'll provide pre-defined keys such as "max", "min",
> >> >>    "byte_width" and "distinct_count" but users can also use
> >> >>    application specific keys.
> >> >>
> >> >> 2. If true, then the value is approximate or best-effort.
> >> >>
> >> >> VALUE_SCHEMA is a dense union with members:
> >> >>
> >> >> | Field Name | Field Type                       | Comments |
> >> >> |------------|----------------------------------| -------- |
> >> >> | int64      | int64                            |          |
> >> >> | uint64     | uint64                           |          |
> >> >> | float64    | float64                          |          |
> >> >> | value      | The same type of the ArrowSchema | (3)      |
> >> >> |            | that is belonged to.             |          |
> >> >>
> >> >> 3. If the ArrowSchema's type is string, this type is also string.
> >> >>
> >> >>    TODO: Is "value" good name? If we refer it from the
> >> >>    top-level statistics schema, we need to use
> >> >>    "value.value". It's a bit strange...
> >> >>
> >> >>
> >> >> What do you think about this proposal? Could you share your
> >> >> comments?
> >> >>
> >> >>
> >> >> Thanks,
> >> >> --
> >> >> kou
> >>
>

Re: [DISCUSS] Statistics through the C data interface

Posted by Sutou Kouhei <ko...@clear-code.com>.
Hi,

>            for non-standard statistics from open-source products the key=0
> combined with string label is the way to go

Where do we store the string label?

I think that we're considering the following schema:

>> map<
>>   // The column index or null if the statistics refer to whole table or batch.
>>   column: int32,
>>   // Statistics key is int32.
>>   // Different keys are assigned for exact value and
>>   // approximate value.
>>   map<int32, dense_union<...needed types based on stat kinds in the keys...>>
>> >

Are you considering the following schema for key=0 case?

map<struct<int32, utf8>,
    dense_union<...needed types based on stat kinds in the keys...>>


Thanks,
-- 
kou

In <CA...@mail.gmail.com>
  "Re: [DISCUSS] Statistics through the C data interface" on Mon, 1 Jul 2024 11:58:44 -0300,
  Felipe Oliveira Carvalho <fe...@gmail.com> wrote:

> Hi,
> 
> You can promise that well-known int32 statistic keys won't ever be higher
> than a certain value (2^18) [1] like TCP IP ports (well-known ports in [0,
> 2^10)) but for non-standard statistics from open-source products the key=0
> combined with string label is the way to go, otherwise collisions would be
> inevitable and everyone would hate us for having integer keys.
> 
> This is not a very weird proposal from my part because integer keys
> representing labels are common in most low-level standardized C APIs (e.g.
> linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
> map all these keys to strings, but with them we keep the "C Data Interface"
> low-level and portable as it should be.
> 
> --
> Felipe
> 
> [1] 2^16 is too small. For instance, OpenGL constants can't be enums
> because C limits enum to 2^16 and that is *not enough*.
> 
> On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
> 
>> Hi,
>>
>> Here is an updated summary so far:
>>
>> ----
>> Use cases:
>>
>> * Optimize query plan: e.g. JOIN for DuckDB
>>
>> Out of scope:
>>
>> * Transmit statistics through not the C data interface
>>   Examples:
>>   * Transmit statistics through Apache Arrow IPC file
>>   * Transmit statistics through Apache Arrow Flight
>> * Multi-column statistics
>> * Constraints information
>> * Indexes information
>>
>> Discussing approach:
>>
>> Standardize Apache Arrow schema for statistics and transmit
>> statistics via separated API call that uses the C data
>> interface.
>>
>> This also works for per-batch statistics.
>>
>> Candidate schema:
>>
>> map<
>>   // The column index or null if the statistics refer to whole table or
>> batch.
>>   column: int32,
>>   // Statistics key is int32.
>>   // Different keys are assigned for exact value and
>>   // approximate value.
>>   map<int32, dense_union<...needed types based on stat kinds in the
>> keys...>>
>> >
>>
>> Discussions:
>>
>> 1. Can we use int32 for statistic keys?
>>    Should we use utf8 (or dictionary<int32, utf8>) for
>>    statistic keys?
>> 2. Hot to support non-standard (vendor-specific)
>>    statistic keys?
>> ----
>>
>> Here is my idea:
>>
>> 1. We can use int32 for statistic keys.
>> 2. We can reserve a specific range for non-standard
>>    statistic keys. Prerequisites of this:
>>    * There is no use case to merge some statistics for
>>      the same data.
>>    * We can't merge statistics for different data.
>>
>> If the prerequisites aren't satisfied:
>>
>> 1. We should use utf8 (or dictionary<int32, utf8>) for
>>    statistic keys?
>> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>>    standard statistic keys or use prefix such as
>>    "vendor1:"/"vendor1." for non-standard statistic keys.
>>
>> Here is Felipe's idea:
>> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>>
>> 1. We can use int32 for statistic keys.
>> 2. We can use the special statistic key + a string identifier
>>    for non-standard statistic keys.
>>
>>
>> What do you think about this?
>>
>>
>> Thanks,
>> --
>> kou
>>
>> In <20...@clear-code.com>
>>   "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
>> 2024 18:27:27 +0900 (JST),
>>   Sutou Kouhei <ko...@clear-code.com> wrote:
>>
>> > Hi,
>> >
>> > Thanks for sharing your comments. Here is a summary so far:
>> >
>> > ----
>> >
>> > Use cases:
>> >
>> > * Optimize query plan: e.g. JOIN for DuckDB
>> >
>> > Out of scope:
>> >
>> > * Transmit statistics through not the C data interface
>> >   Examples:
>> >   * Transmit statistics through Apache Arrow IPC file
>> >   * Transmit statistics through Apache Arrow Flight
>> >
>> > Candidate approaches:
>> >
>> > 1. Pass statistics (encoded as an Apache Arrow data) via
>> >    ArrowSchema metadata
>> >    * This embeds statistics address into metadata
>> >    * It's for avoiding using Apache Arrow IPC format with
>> >      the C data interface
>> > 2. Embed statistics (encoded as an Apache Arrow data) into
>> >    ArrowSchema metadata
>> >    * This adds statistics to metadata in Apache Arrow IPC
>> >      format
>> > 3. Embed statistics (encoded as JSON) into ArrowArray
>> >    metadata
>> > 4. Standardize Apache Arrow schema for statistics and
>> >    transmit statistics via separated API call that uses the
>> >    C data interface
>> > 5. Use ADBC
>> >
>> > ----
>> >
>> > I think that 4. is the best approach in these candidates.
>> >
>> > 1. Embedding statistics address is tricky.
>> > 2. Consumers need to parse Apache Arrow IPC format data.
>> >    (The C data interface consumers may not have the
>> >    feature.)
>> > 3. This will work but 4. is more generic.
>> > 5. ADBC is too large to use only for statistics.
>> >
>> > What do you think about this?
>> >
>> >
>> > If we select 4., we need to standardize Apache Arrow schema
>> > for statistics. How about the following schema?
>> >
>> > ----
>> > Metadata:
>> >
>> > | Name                       | Value | Comments |
>> > |----------------------------|-------|--------- |
>> > | ARROW::statistics::version | 1.0.0 | (1)      |
>> >
>> > (1) This follows semantic versioning.
>> >
>> > Fields:
>> >
>> > | Name           | Type                  | Comments |
>> > |----------------|-----------------------| -------- |
>> > | column         | utf8                  | (2)      |
>> > | key            | utf8 not null         | (3)      |
>> > | value          | VALUE_SCHEMA not null |          |
>> > | is_approximate | bool not null         | (4)      |
>> >
>> > (2) If null, then the statistic applies to the entire table.
>> >     It's for "row_count".
>> > (3) We'll provide pre-defined keys such as "max", "min",
>> >     "byte_width" and "distinct_count" but users can also use
>> >     application specific keys.
>> > (4) If true, then the value is approximate or best-effort.
>> >
>> > VALUE_SCHEMA is a dense union with members:
>> >
>> > | Name    | Type    |
>> > |---------|---------|
>> > | int64   | int64   |
>> > | uint64  | uint64  |
>> > | float64 | float64 |
>> > | binary  | binary  |
>> >
>> > If a column is an int32 column, it uses int64 for
>> > "max"/"min". We don't provide all types here. Users should
>> > use a compatible type (int64 for a int32 column) instead.
>> > ----
>> >
>> >
>> > Thanks,
>> > --
>> > kou
>> >
>> >
>> > In <20...@clear-code.com>
>> >   "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
>> 2024 11:37:08 +0900 (JST),
>> >   Sutou Kouhei <ko...@clear-code.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> We're discussing how to provide statistics through the C
>> >> data interface at:
>> >> https://github.com/apache/arrow/issues/38837
>> >>
>> >> If you're interested in this feature, could you share your
>> >> comments?
>> >>
>> >>
>> >> Motivation:
>> >>
>> >> We can interchange Apache Arrow data by the C data interface
>> >> in the same process. For example, we can pass Apache Arrow
>> >> data read by Apache Arrow C++ (provider) to DuckDB
>> >> (consumer) through the C data interface.
>> >>
>> >> A provider may know Apache Arrow data statistics. For
>> >> example, a provider can know statistics when it reads Apache
>> >> Parquet data because Apache Parquet may provide statistics.
>> >>
>> >> But a consumer can't know statistics that are known by a
>> >> producer. Because there isn't a standard way to provide
>> >> statistics through the C data interface. If a consumer can
>> >> know statistics, it can process Apache Arrow data faster
>> >> based on statistics.
>> >>
>> >>
>> >> Proposal:
>> >>
>> >> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
>> >>
>> >> How about providing statistics as a metadata in ArrowSchema?
>> >>
>> >> We reserve "ARROW" namespace for internal Apache Arrow use:
>> >>
>> >>
>> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
>> >>
>> >>> The ARROW pattern is a reserved namespace for internal
>> >>> Arrow use in the custom_metadata fields. For example,
>> >>> ARROW:extension:name.
>> >>
>> >> So we can use "ARROW:statistics" for the metadata key.
>> >>
>> >> We can represent statistics as a ArrowArray like ADBC does.
>> >>
>> >> Here is an example ArrowSchema that is for a record batch
>> >> that has "int32 column1" and "string column2":
>> >>
>> >> ArrowSchema {
>> >>   .format = "+siu",
>> >>   .metadata = {
>> >>     "ARROW:statistics" => ArrowArray*, /* table-level statistics such
>> as row count */
>> >>   },
>> >>   .children = {
>> >>     ArrowSchema {
>> >>       .name = "column1",
>> >>       .format = "i",
>> >>       .metadata = {
>> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>> such as count distinct */
>> >>       },
>> >>     },
>> >>     ArrowSchema {
>> >>       .name = "column2",
>> >>       .format = "u",
>> >>       .metadata = {
>> >>         "ARROW:statistics" => ArrowArray*, /* column-level statistics
>> such as count distinct */
>> >>       },
>> >>     },
>> >>   },
>> >> }
>> >>
>> >> The metadata value (ArrowArray* part) of '"ARROW:statistics"
>> >> => ArrowArray*' is a base 10 string of the address of the
>> >> ArrowArray. Because we can use only string for metadata
>> >> value. You can't release the statistics ArrowArray*. (Its
>> >> release is a no-op function.) It follows
>> >>
>> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
>> >> semantics. (The base ArrowSchema owns statistics
>> >> ArrowArray*.)
>> >>
>> >>
>> >> ArrowArray* for statistics use the following schema:
>> >>
>> >> | Field Name     | Field Type                       | Comments |
>> >> |----------------|----------------------------------| -------- |
>> >> | key            | string not null                  | (1)      |
>> >> | value          | `VALUE_SCHEMA` not null          |          |
>> >> | is_approximate | bool not null                    | (2)      |
>> >>
>> >> 1. We'll provide pre-defined keys such as "max", "min",
>> >>    "byte_width" and "distinct_count" but users can also use
>> >>    application specific keys.
>> >>
>> >> 2. If true, then the value is approximate or best-effort.
>> >>
>> >> VALUE_SCHEMA is a dense union with members:
>> >>
>> >> | Field Name | Field Type                       | Comments |
>> >> |------------|----------------------------------| -------- |
>> >> | int64      | int64                            |          |
>> >> | uint64     | uint64                           |          |
>> >> | float64    | float64                          |          |
>> >> | value      | The same type of the ArrowSchema | (3)      |
>> >> |            | that is belonged to.             |          |
>> >>
>> >> 3. If the ArrowSchema's type is string, this type is also string.
>> >>
>> >>    TODO: Is "value" good name? If we refer it from the
>> >>    top-level statistics schema, we need to use
>> >>    "value.value". It's a bit strange...
>> >>
>> >>
>> >> What do you think about this proposal? Could you share your
>> >> comments?
>> >>
>> >>
>> >> Thanks,
>> >> --
>> >> kou
>>

Re: [DISCUSS] Statistics through the C data interface

Posted by Antoine Pitrou <an...@python.org>.
Hmmm, I strive to understand why a `(int32, utf8)` tuple for statistic 
keys would be any simpler to implement than either `int32` *or* `utf8` 
*or* `dictionary(int32, utf8)`.

Let's keep in mind that we would like to keep things simple for 
consumers and producers of statistics.

We should also ask ourselves what the typical cardinality would be.

* Cardinality of columns can be very high (thousands or more?);
* Cardinality of statistic kinds/keys is bound to remain low (how many 
different kinds statistics can we reasonably devise that would be useful 
to transmit over Arrow).

Regards

Antoine.


Le 01/07/2024 à 16:58, Felipe Oliveira Carvalho a écrit :
> Hi,
> 
> You can promise that well-known int32 statistic keys won't ever be higher
> than a certain value (2^18) [1] like TCP IP ports (well-known ports in [0,
> 2^10)) but for non-standard statistics from open-source products the key=0
> combined with string label is the way to go, otherwise collisions would be
> inevitable and everyone would hate us for having integer keys.
> 
> This is not a very weird proposal from my part because integer keys
> representing labels are common in most low-level standardized C APIs (e.g.
> linux syscalls, ioctls, OpenGL, Vulcan...). I expect higher level APIs to
> map all these keys to strings, but with them we keep the "C Data Interface"
> low-level and portable as it should be.
> 
> --
> Felipe
> 
> [1] 2^16 is too small. For instance, OpenGL constants can't be enums
> because C limits enum to 2^16 and that is *not enough*.
> 
> On Thu, Jun 20, 2024 at 7:43 AM Sutou Kouhei <ko...@clear-code.com> wrote:
> 
>> Hi,
>>
>> Here is an updated summary so far:
>>
>> ----
>> Use cases:
>>
>> * Optimize query plan: e.g. JOIN for DuckDB
>>
>> Out of scope:
>>
>> * Transmit statistics through not the C data interface
>>    Examples:
>>    * Transmit statistics through Apache Arrow IPC file
>>    * Transmit statistics through Apache Arrow Flight
>> * Multi-column statistics
>> * Constraints information
>> * Indexes information
>>
>> Discussing approach:
>>
>> Standardize Apache Arrow schema for statistics and transmit
>> statistics via separated API call that uses the C data
>> interface.
>>
>> This also works for per-batch statistics.
>>
>> Candidate schema:
>>
>> map<
>>    // The column index or null if the statistics refer to whole table or
>> batch.
>>    column: int32,
>>    // Statistics key is int32.
>>    // Different keys are assigned for exact value and
>>    // approximate value.
>>    map<int32, dense_union<...needed types based on stat kinds in the
>> keys...>>
>>>
>>
>> Discussions:
>>
>> 1. Can we use int32 for statistic keys?
>>     Should we use utf8 (or dictionary<int32, utf8>) for
>>     statistic keys?
>> 2. Hot to support non-standard (vendor-specific)
>>     statistic keys?
>> ----
>>
>> Here is my idea:
>>
>> 1. We can use int32 for statistic keys.
>> 2. We can reserve a specific range for non-standard
>>     statistic keys. Prerequisites of this:
>>     * There is no use case to merge some statistics for
>>       the same data.
>>     * We can't merge statistics for different data.
>>
>> If the prerequisites aren't satisfied:
>>
>> 1. We should use utf8 (or dictionary<int32, utf8>) for
>>     statistic keys?
>> 2. We can use reserved prefix such as "ARROW:"/"arrow." for
>>     standard statistic keys or use prefix such as
>>     "vendor1:"/"vendor1." for non-standard statistic keys.
>>
>> Here is Felipe's idea:
>> https://lists.apache.org/thread/gr2nmlrwr7d5wkz3zgq6vy5q0ow8xof2
>>
>> 1. We can use int32 for statistic keys.
>> 2. We can use the special statistic key + a string identifier
>>     for non-standard statistic keys.
>>
>>
>> What do you think about this?
>>
>>
>> Thanks,
>> --
>> kou
>>
>> In <20...@clear-code.com>
>>    "Re: [DISCUSS] Statistics through the C data interface" on Thu, 06 Jun
>> 2024 18:27:27 +0900 (JST),
>>    Sutou Kouhei <ko...@clear-code.com> wrote:
>>
>>> Hi,
>>>
>>> Thanks for sharing your comments. Here is a summary so far:
>>>
>>> ----
>>>
>>> Use cases:
>>>
>>> * Optimize query plan: e.g. JOIN for DuckDB
>>>
>>> Out of scope:
>>>
>>> * Transmit statistics through not the C data interface
>>>    Examples:
>>>    * Transmit statistics through Apache Arrow IPC file
>>>    * Transmit statistics through Apache Arrow Flight
>>>
>>> Candidate approaches:
>>>
>>> 1. Pass statistics (encoded as an Apache Arrow data) via
>>>     ArrowSchema metadata
>>>     * This embeds statistics address into metadata
>>>     * It's for avoiding using Apache Arrow IPC format with
>>>       the C data interface
>>> 2. Embed statistics (encoded as an Apache Arrow data) into
>>>     ArrowSchema metadata
>>>     * This adds statistics to metadata in Apache Arrow IPC
>>>       format
>>> 3. Embed statistics (encoded as JSON) into ArrowArray
>>>     metadata
>>> 4. Standardize Apache Arrow schema for statistics and
>>>     transmit statistics via separated API call that uses the
>>>     C data interface
>>> 5. Use ADBC
>>>
>>> ----
>>>
>>> I think that 4. is the best approach in these candidates.
>>>
>>> 1. Embedding statistics address is tricky.
>>> 2. Consumers need to parse Apache Arrow IPC format data.
>>>     (The C data interface consumers may not have the
>>>     feature.)
>>> 3. This will work but 4. is more generic.
>>> 5. ADBC is too large to use only for statistics.
>>>
>>> What do you think about this?
>>>
>>>
>>> If we select 4., we need to standardize Apache Arrow schema
>>> for statistics. How about the following schema?
>>>
>>> ----
>>> Metadata:
>>>
>>> | Name                       | Value | Comments |
>>> |----------------------------|-------|--------- |
>>> | ARROW::statistics::version | 1.0.0 | (1)      |
>>>
>>> (1) This follows semantic versioning.
>>>
>>> Fields:
>>>
>>> | Name           | Type                  | Comments |
>>> |----------------|-----------------------| -------- |
>>> | column         | utf8                  | (2)      |
>>> | key            | utf8 not null         | (3)      |
>>> | value          | VALUE_SCHEMA not null |          |
>>> | is_approximate | bool not null         | (4)      |
>>>
>>> (2) If null, then the statistic applies to the entire table.
>>>      It's for "row_count".
>>> (3) We'll provide pre-defined keys such as "max", "min",
>>>      "byte_width" and "distinct_count" but users can also use
>>>      application specific keys.
>>> (4) If true, then the value is approximate or best-effort.
>>>
>>> VALUE_SCHEMA is a dense union with members:
>>>
>>> | Name    | Type    |
>>> |---------|---------|
>>> | int64   | int64   |
>>> | uint64  | uint64  |
>>> | float64 | float64 |
>>> | binary  | binary  |
>>>
>>> If a column is an int32 column, it uses int64 for
>>> "max"/"min". We don't provide all types here. Users should
>>> use a compatible type (int64 for a int32 column) instead.
>>> ----
>>>
>>>
>>> Thanks,
>>> --
>>> kou
>>>
>>>
>>> In <20...@clear-code.com>
>>>    "[DISCUSS] Statistics through the C data interface" on Wed, 22 May
>> 2024 11:37:08 +0900 (JST),
>>>    Sutou Kouhei <ko...@clear-code.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We're discussing how to provide statistics through the C
>>>> data interface at:
>>>> https://github.com/apache/arrow/issues/38837
>>>>
>>>> If you're interested in this feature, could you share your
>>>> comments?
>>>>
>>>>
>>>> Motivation:
>>>>
>>>> We can interchange Apache Arrow data by the C data interface
>>>> in the same process. For example, we can pass Apache Arrow
>>>> data read by Apache Arrow C++ (provider) to DuckDB
>>>> (consumer) through the C data interface.
>>>>
>>>> A provider may know Apache Arrow data statistics. For
>>>> example, a provider can know statistics when it reads Apache
>>>> Parquet data because Apache Parquet may provide statistics.
>>>>
>>>> But a consumer can't know statistics that are known by a
>>>> producer. Because there isn't a standard way to provide
>>>> statistics through the C data interface. If a consumer can
>>>> know statistics, it can process Apache Arrow data faster
>>>> based on statistics.
>>>>
>>>>
>>>> Proposal:
>>>>
>>>> https://github.com/apache/arrow/issues/38837#issuecomment-2123728784
>>>>
>>>> How about providing statistics as a metadata in ArrowSchema?
>>>>
>>>> We reserve "ARROW" namespace for internal Apache Arrow use:
>>>>
>>>>
>> https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata
>>>>
>>>>> The ARROW pattern is a reserved namespace for internal
>>>>> Arrow use in the custom_metadata fields. For example,
>>>>> ARROW:extension:name.
>>>>
>>>> So we can use "ARROW:statistics" for the metadata key.
>>>>
>>>> We can represent statistics as a ArrowArray like ADBC does.
>>>>
>>>> Here is an example ArrowSchema that is for a record batch
>>>> that has "int32 column1" and "string column2":
>>>>
>>>> ArrowSchema {
>>>>    .format = "+siu",
>>>>    .metadata = {
>>>>      "ARROW:statistics" => ArrowArray*, /* table-level statistics such
>> as row count */
>>>>    },
>>>>    .children = {
>>>>      ArrowSchema {
>>>>        .name = "column1",
>>>>        .format = "i",
>>>>        .metadata = {
>>>>          "ARROW:statistics" => ArrowArray*, /* column-level statistics
>> such as count distinct */
>>>>        },
>>>>      },
>>>>      ArrowSchema {
>>>>        .name = "column2",
>>>>        .format = "u",
>>>>        .metadata = {
>>>>          "ARROW:statistics" => ArrowArray*, /* column-level statistics
>> such as count distinct */
>>>>        },
>>>>      },
>>>>    },
>>>> }
>>>>
>>>> The metadata value (ArrowArray* part) of '"ARROW:statistics"
>>>> => ArrowArray*' is a base 10 string of the address of the
>>>> ArrowArray. Because we can use only string for metadata
>>>> value. You can't release the statistics ArrowArray*. (Its
>>>> release is a no-op function.) It follows
>>>>
>> https://arrow.apache.org/docs/format/CDataInterface.html#member-allocation
>>>> semantics. (The base ArrowSchema owns statistics
>>>> ArrowArray*.)
>>>>
>>>>
>>>> ArrowArray* for statistics use the following schema:
>>>>
>>>> | Field Name     | Field Type                       | Comments |
>>>> |----------------|----------------------------------| -------- |
>>>> | key            | string not null                  | (1)      |
>>>> | value          | `VALUE_SCHEMA` not null          |          |
>>>> | is_approximate | bool not null                    | (2)      |
>>>>
>>>> 1. We'll provide pre-defined keys such as "max", "min",
>>>>     "byte_width" and "distinct_count" but users can also use
>>>>     application specific keys.
>>>>
>>>> 2. If true, then the value is approximate or best-effort.
>>>>
>>>> VALUE_SCHEMA is a dense union with members:
>>>>
>>>> | Field Name | Field Type                       | Comments |
>>>> |------------|----------------------------------| -------- |
>>>> | int64      | int64                            |          |
>>>> | uint64     | uint64                           |          |
>>>> | float64    | float64                          |          |
>>>> | value      | The same type of the ArrowSchema | (3)      |
>>>> |            | that is belonged to.             |          |
>>>>
>>>> 3. If the ArrowSchema's type is string, this type is also string.
>>>>
>>>>     TODO: Is "value" good name? If we refer it from the
>>>>     top-level statistics schema, we need to use
>>>>     "value.value". It's a bit strange...
>>>>
>>>>
>>>> What do you think about this proposal? Could you share your
>>>> comments?
>>>>
>>>>
>>>> Thanks,
>>>> --
>>>> kou
>>
>