Re: [DISCUSS] Add arrow.range canonical extension type for bounded ranges

[Rok Mihevc <ro...@gmail.com>]

>
>  > Current proposal has a "closed" parameter with {left, right, both,
>  > neither}. How about we extend that to {left, right, both, neither,
>  > left_per_value, right_per_value, both_per_value}?
>  > This is similar to proposal 3.
>
> The asymmetric sub-modes are a nice space optimization when only
> one side varies, but my intuition is that this is a rare case. Discrete
> ranges canonicalize on both sides, and continuous ranges typically vary
> on both. Do you have a concrete use-case in mind?
>

I don't have a specific use case. However if we specify the more
general option here, does it increase implementation complexity or runtime
memory requirements? If not I'd rather we specify a more general solution.

 > Should we specify closedness information per value as optional?
>  > Could null closedness have some meaning?
>
> Do you mean that the storage type should be `Struct<lower: T, upper: T,
> lower_inc: bool (nullable), upper_inc: bool (nullable)>`?
> I guess the meaning would be "closedness is unspecified for this value,
> fall back to the default closedness specified in column-metadata"?
> I would not do that, because this means carrying two extra validity
> bitmaps just to encode a fallback. Copying the default closedness
> into the flags is faster and more memory efficient, as the
> flags stay non-nullable, and uniform columns compress trivially under
> RLE or dictionary encoding.
> Or did I misunderstand your proposal?
>

I like that interpretation. Perhaps we can allow for both nullable and
non-nullable options?

Rok

Re: [DISCUSS] Add arrow.range canonical extension type for bounded ranges

[Hoeze <gi...@frhoelzlwimmer.de>]

> I don't have a specific use case. However if we specify the more
> general option here, does it increase implementation complexity or runtime
> memory requirements? If not I'd rather we specify a more general solution.
I think it does, mainly on the complexity side rather than memory.

Per-value closedness as a sub-mode of `arrow.range`'s `closed` enum is essentially option 3. We moved away from that towards a separate `arrow.range_inc` type so the storage layout is determined by the type name rather than by JSON metadata, as this is easier for downstream tooling to get right. Folding `per_value` back into the `closed` enum would reintroduce that coupling.

Moreover, the asymmetric variants (`left_per_value`, `right_per_value`) would add a third storage layout with a single flag column, so implementations would have to handle three cases instead of one-name-to-one-layout. That only saves a single boolean column, which is bit-packed and compresses to almost nothing under RLE when uniform, so the extra generality doesn't really pay for itself.

Would you be OK keeping the two types separate, or do you feel strongly about a single type with a `per_value` mode?

> I like that interpretation. Perhaps we can allow for both nullable and
> non-nullable options?
I think a nullable flag with a fallback ends up equivalent to just storing the resolved value. Whatever default we'd fall back to, we can write that directly into the non-nullable flag. Same information, no extra validity bitmap, and it still compresses perfectly under RLE/dictionary when uniform. So I'd lean towards keeping the flags non-nullable and required, unless you see a case that this approach misses.

Best,
Hoeze

Am 02.06.26 um 18:42 schrieb Rok Mihevc:
>>   > Current proposal has a "closed" parameter with {left, right, both,
>>   > neither}. How about we extend that to {left, right, both, neither,
>>   > left_per_value, right_per_value, both_per_value}?
>>   > This is similar to proposal 3.
>>
>> The asymmetric sub-modes are a nice space optimization when only
>> one side varies, but my intuition is that this is a rare case. Discrete
>> ranges canonicalize on both sides, and continuous ranges typically vary
>> on both. Do you have a concrete use-case in mind?
>>
> I don't have a specific use case. However if we specify the more
> general option here, does it increase implementation complexity or runtime
> memory requirements? If not I'd rather we specify a more general solution.
>
>   > Should we specify closedness information per value as optional?
>>   > Could null closedness have some meaning?
>>
>> Do you mean that the storage type should be `Struct<lower: T, upper: T,
>> lower_inc: bool (nullable), upper_inc: bool (nullable)>`?
>> I guess the meaning would be "closedness is unspecified for this value,
>> fall back to the default closedness specified in column-metadata"?
>> I would not do that, because this means carrying two extra validity
>> bitmaps just to encode a fallback. Copying the default closedness
>> into the flags is faster and more memory efficient, as the
>> flags stay non-nullable, and uniform columns compress trivially under
>> RLE or dictionary encoding.
>> Or did I misunderstand your proposal?
>>
> I like that interpretation. Perhaps we can allow for both nullable and
> non-nullable options?
>
> Rok
>