You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by "Florian R. Hölzlwimmer" <gi...@frhoelzlwimmer.de> on 2026/05/24 23:10:37 UTC

[DISCUSS] Add arrow.range canonical extension type for bounded ranges

Hi all,

Following a suggestion from @rok on GitHub, I'd like to open a discussion about adding a canonical extension type for bounded ranges to Arrow.

Background
==========

So far, Arrow has no canonical way to represent a bounded range (a mathematical interval with a lower and an upper endpoint), e.g. a numeric range `[0, 10)`, a date range, or a timestamp period. Today such data is modeled ad hoc with two separate columns or with system-specific extension types, which hurts interoperability. A canonical range type will be useful to libraries like Pandas, Polars/Polars-bio, IRanges/PyRanges, database connectors, ...

This is distinct from Arrow's existing calendar `Interval` type (`INTERVAL_MONTHS` / `INTERVAL_DAY_TIME` / `INTERVAL_MONTH_DAY_NANO`), which represents a duration (a signed amount of time), not a bounded set. Databases like PostgreSQL make the same distinction: SQL uses `INTERVAL` for durations and `RANGE` / `PERIOD` for bounded sets. This proposal follows that convention by naming the type `arrow.range`.


Proposed design
===============

   * Extension name: `arrow.range`.

   * Storage type: `Struct<lower: T, upper: T>`. When subtype `T` is nullable, a null bound represents an unbounded (infinite) endpoint.

       * Field names `lower` / `upper` follow PostgreSQL's convention for ordering clarity (Pandas uses `left` / `right`).
       * The subtype `T` may be any orderable Arrow type (the numeric, temporal and decimal families, etc.). Nested or non-comparable types are out of scope.

   * Metadata: a JSON object `{"closed": "..."}` where `closed` is one of `left`, `right`, `both`, `neither` (pandas vocabulary; `left` = lower inclusive / upper exclusive, etc.). Required on the wire so a serialized `arrow.range` is always unambiguous. Unknown JSON keys are ignored for forward compatibility.

   * A range is empty implicitly when `lower > upper`, or when `lower == upper` with at least one bound exclusive. A range with `lower > upper` is therefore valid (it denotes the empty set), not an error.

This mirrors pandas' interval support closely enough that `arrow.range` would give `pandas.IntervalArray` / `IntervalIndex` a natural, lossless Arrow representation for round-tripping.


References
==========

   - Full proposal and rationale:
     https://github.com/apache/arrow/issues/50027
   - Draft C++/Python implementation:
     https://github.com/apache/arrow/pull/50028

I'd appreciate any feedback on the overall direction and on the specific design choices: field naming (`lower`/`upper` vs. `left`/`right`), the `closed` parameter, the treatment of unbounded endpoints via nullability, and the set of supported subtypes.

Many thanks,
Hoeze

Re: [DISCUSS] Add arrow.range canonical extension type for bounded ranges

Posted by gi...@frhoelzlwimmer.de.

Hi all,

a gentle ping on this thread.

tl;dr of my last mail: Separate types per closedness do not
make kernel dispatch simpler. Arrow's kernel matching can
already see extension type parameters, so one type with a
`closed` parameter gets the same "no kernel matching input
types" safety. This is how match::TimestampTypeUnit already
works for timestamps.

Proposal:
1. One range type with a `closed` parameter.
2. One per-value type for continuous ranges.
3. Names: FixedClosednessRangeType / VariableClosednessRangeType
    (like FixedShapeTensorType / VariableShapeTensorType),
    unless someone has a better idea.

The spec text in the Pull-Request <https://github.com/apache/arrow/pull/50028/> now follows this proposal:
https://github.com/Hoeze/arrow/blob/5ff2abdc7fec30a771c1ec8f4f4e53cf6d1ba00a/docs/source/format/CanonicalExtensions.rst#fixed-closedness-range

@Felipe: Are you fine with this?
If there are no objections by Monday, Sep 28, I would assume
lazy consensus, update the second implementation in arrow-rs,
and start a [VOTE] thread.

Best,
Florian

On 12.09.26 17:47, git.ich@frhoelzlwimmer.de wrote:
> Hi Felipe,
>
> > I feel like you should have different types depending on whether a
> > bound is open or not.
>
> I found your idea tempting and was convinced by it for a while. But
> after looking into how Arrow C++ actually resolves kernels, I now lean
> the other way.
>
> > Routing compute functions based on metadata would be hellish and would
> > force compute function writers to care about all possible values of
> > the metadata when a new function on ranges is introduced.
>
> I checked this, and separate types help less than it seems. All
> extension types report `Type::EXTENSION`, so the kernel matcher cannot
> distinguish `half_open_range` from `closed_range` on its own. You still
> need a custom matcher on the extension name. That is the same work as
> matching one `arrow.range` and reading its `closed` flag.
>
> Also, `closed` is not parsed at compute time. It is read once when the
> type is reconstructed. After that it is a typed enum accessor,
> `RangeType::closed()`, just like reading a decimal's scale.
>
> Where you are right: If a function author does not account for
> closedness at all, separate types fail more safely. Passing a
> `closed_range` to a half-open-only kernel would give a loud "no kernel
> matching input types", while a single parameterized type can be
> silently mishandled.
>
> However, I believe that there is a better fix than having more types:
>
> 1. An author handles closedness but forgets a case:
>
>    That is caught at compile time. A switch over the four `RangeClosed`
>    values with no default becomes a compile error if a case is missing
>    (under -Wswitch/-Werror). The existing `ClosedToString` already
>    relies on this.
>
>    Four separate types give less protection here, because forgetting to
>    register the kernel for `closed_range` would still compile and only
>    fail at runtime.
>
> 2. An author never reads `closed` at all:
>
>    In your scheme that author registers `contains()` for
>    `half_open_range` only, and a `closed_range` input is rejected with
>    "no kernel matching input types".
>
>    My parameter design would give the same protection through the
>    kernel's declared input type. A kernel registered with a
>    `match::RangeClosed(Left)` matcher is only picked for `arrow.range`
>    arrays whose `closed == Left`, and a `closed == Both` input is
>    rejected with the same error.
>    This is the same way `match::TimestampTypeUnit` works already.
>
> > With the different types, you could have a canonicalization layer
> > that allocates a new array before a function learns to handle all
> > possible range types.
>
> That layer fits on top of the matcher and does not need separate types.
> A `canonicalize_to_half_open(array)` helper can convert any discrete
> `arrow.range` to `closed == Left`, and a kernel author can call it
> instead of rejecting the input.
>
> However, please note that canonicalizing to half-open does not work for
> continuous subtypes, as discussed earlier in this thread. Technically
> every Arrow type has a smallest representable increment (e.g. `[1, 5]`
> over `double` would become `[1, nextafter(5))`). But such a conversion
> would change the bound, the result would differ after a lossless cast
> between precisions or timestamp units, and a round trip through
> PostgreSQL's `numrange` would keep the converted
> `[1, 5.000000000000001)` instead of restoring `[1, 5]`.
>
> This applies to both designs. A half-open-only `contains()` over
> `numrange` must handle open and closed bounds directly either way.
>
> > Making the openness per value will complicate columnar compute
> > functions, even though it simplifies transport from databases like
> > PostgreSQL.
>
> I agree, but per-value closedness is the only lossless option for
> continuous ranges. PostgreSQL stores the inclusive/exclusive flags per
> value, and one `numrange` column can legitimately hold both (1, 5) and
> [1, 5] values. None of the four fixed types can represent such a
> mixed-closedness column. So a per-value type has to exist regardless of
> what we do with the fixed case.
>
> > At the same time, expecting application consumers of an ADBC driver to
> > check the open/closed flags per value is a big ask.
>
> Here I disagree. In a `range_inc` column the inclusivity varies per row.
> A consumer that cares about inclusivity has to read it per row in any
> design, because that is where the information is.
>
> Applications that care about per-value closedness already do this
> (e.g. temporal intervals, coordinates). A consumer that does not care
> can cast the column to the fixed-closedness `arrow.range` and drop
> the flags.
>
> One more thing: Four fixed types are quite unidiomatic for Arrow. They
> would have identical storage, `Struct<lower, upper>`, and differ only
> in a semantic flag. Arrow normally expresses that as a parameter on one
> type rather than a separate type per value (`decimal` carries scale and
> precision, `timestamp` carries unit and time zone). It would also mean
> five canonical types to specify and maintain instead of two, across
> every implementation: C++, Rust, Go, Java, JS, Swift, nanoarrow, and so
> on.
>
> Bottom line, I would prefer keeping one `arrow.range` type with a
> `closed` parameter (the same shape as pandas' IntervalArray) plus one
> per-value type. What do you think?
>
> Best,
> Hoeze
>
>
> PS: What are the next steps once this discussion has converged?
>
>
> On 11.09.26 02:28, git.ich@frhoelzlwimmer.de wrote:
>> Dear Rok, dear Felipe,
>>
>> yes, I'm definitely interested in getting it over the finish line :)
>>
>> @Felipe I'm very sorry, somehow your message slipped my attention.
>> Let me sleep over it, and I'll give you a founded response tomorrow ^^
>>
>> Best,
>> Florian
>>
>>
>> On 10.09.26 18:49, Rok Mihevc wrote:
>>> Hey all,
>>>
>>> Is there still interest in standardizing this? I thought we were on a good
>>> track and it'd be nice to complete it.
>>>
>>> Best,
>>> Rok
>>>
>>>
>>>
>>> On Mon, Jun 8, 2026 at 7:26 PM Felipe Oliveira Carvalho <fe...@gmail.com>
>>> wrote:
>>>
>>>> I feel like you should have different types depending on whether a bound is
>>>> open or not.
>>>>
>>>> [.., ..) arrow.half_open_range
>>>> (.., ..) arrow.closed_range
>>>> (.., ..] arrow.lower_half_open_range
>>>> [.., ..] arrow.open_range
>>>>
>>>> Routing compute functions based on metadata would be hellish and would
>>>> force compute function writers to care about all possible values of the
>>>> metadata when a new function on ranges is introduced. With the different
>>>> types, you could have a canonicalization layer that allocates a new array
>>>> before a function learns to handle all possible range types.
>>>>
>>>> Example: a contains() implementation that computes directly only
>>>> arrow.half_open_range arrays, but when that is not the input, it calls a
>>>> utility that can canonicalize every range type to arrow.half_open_range.
>>>>
>>>> Making the openness per value will complicate columnar compute functions,
>>>> even though it simplifies transport from databases like PostgreSQL. You
>>>> would not need to canonicalize the output from a PostgreSQL query into one
>>>> of the range types in the ADBC driver, for instance. At the same time,
>>>> expecting application consumers of an ADBC driver to check the open/closed
>>>> flags per value is a big ask.
>>>>
>>>> -- 
>>>> Felipe
>>>>
>>>> On Sun, Jun 7, 2026 at 11:23 AM Hoeze <gi...@frhoelzlwimmer.de> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> What is your opinion on naming the range types with a fixed closedness
>>>> and
>>>>> a per-value closedness?
>>>>> Currently, they are called `RangeType` and `RangeIncType`, which Rok
>>>>> considers poor naming (and I agree).
>>>>>
>>>>> Some ideas:
>>>>>
>>>>>   1. RangeType | VarRangeType
>>>>>   2. RangeType | PerValueRangeType
>>>>>   3. RangeType | GranularRangeType
>>>>>   4. FixedClosednessRangeType | VariableClosednessRangeType
>>>>>
>>>>> My personal favorite is (4), as it mirrors the tensor type naming and
>>>>> points out the difference.
>>>>> However, it sounds quite long and clumpy.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Best,
>>>>> Hoeze
>>>>>
>>>>> On 04/06/2026 22:30, Hoeze wrote:
>>>>>> Thank you for your feedback, Antoine.
>>>>>>
>>>>>> I updated my draft PR and added the 'range_inc' type:
>>>>>> https://github.com/apache/arrow/pull/50028/
>>>>>>
>>>>>> Please let me know if you have any further suggestions :)
>>>>>>
>>>>>> Best,
>>>>>> Hoeze
>>>>>>
>>>>>> Am 02.06.26 um 19:00 schrieb Antoine Pitrou:
>>>>>>> Le 25/05/2026 à 16:54, Hoeze a écrit :
>>>>>>>> Yes, you're right, the current proposal would probably not be
>>>>>>>> sufficient for continuous PostgreSQL ranges.
>>>>>>>>
>>>>>>>> Column level boundary flags were intentional as it allows to check
>>>>>>>> closedness in the schema instead of during runtime. This is also how
>>>>>>>> Pandas' `IntervalArray`/`IntervalIndex` works.
>>>>>>>>
>>>>>>>> PostgreSQL's built-in discrete ranges (`int4range`, `int8range`,
>>>>>>>> `daterange`) canonicalize to left-closed intervals; here my proposal
>>>>>>>> would be sufficient. However, continuous ranges (`numrange`,
>>>>>>>> `tsrange`, `tstzrange`, ...) cannot be canonicalized. In this case
>>>>>>>> my proposal would indeed not be flexible enough.
>>>>>>>>
>>>>>>>> I could imagine a number of possible solutions to this shortcoming:
>>>>>>>>
>>>>>>>>      * Union type of all four closedness versions:
>>>>>>>>        Possible but not very elegant. Would shift the implementation
>>>>>>>>        burden towards the applications, that have to support union
>>>>>>>>        types.
>>>>>>>>
>>>>>>>>      * Create a separate canonical data type for per-value boundary
>>>>>>>>        flags:
>>>>>>>>        Storage type `Struct<lower: T, upper: T, lower_inc: bool,
>>>>>>>>        upper_inc: bool>`, mirroring PostgreSQL's internal
>>>>>>>>        representation. Both types would coexist: `arrow.range` for the
>>>>>>>>        uniform case (and for canonicalized discrete PostgreSQL
>>>> ranges),
>>>>>>>>        and e.g. `arrow.range_inc` for continuous (PostgreSQL) ranges.
>>>>>>>>
>>>>>>>>      * Extend `arrow.range` itself with a per-value mode:
>>>>>>>>        Keep a single extension type, but allow
>>>>>>>>        `{"closed": "per_value"}` in the metadata, in which case the
>>>>>>>>        storage struct gains two boolean fields `lower_inc` and
>>>>>>>>        `upper_inc`. One extension name, two storage layouts. Simpler
>>>>>>>>        from a type-registry standpoint, slightly more conditional
>>>> logic
>>>>>>>>        in implementations.
>>>>>>>>
>>>>>>>>      * Always store per-value flags:
>>>>>>>>        Drop the metadata key entirely and always use
>>>>>>>>        `Struct<lower: T, upper: T, lower_inc: bool, upper_inc: bool>`.
>>>>>>>>        Two extra bytes per row uncompressed, but highly
>>>> RLE/dictionary-
>>>>>>>>        friendly when uniform (which it usually is). Maximally simple
>>>> to
>>>>>>>>        specify, at the cost of some overhead in the common
>>>> pandas-style
>>>>>>>>        case.
>>>>>>>>
>>>>>>>> I currently lean towards the second option, as it preserves the
>>>>>>>> schema-level check for the common case while still giving continuous,
>>>>>>>> per-value closedness ranges a lossless path. Fixed-shape tensor vs.
>>>>>>>> variable-shape tensor extension types went the same route. The main
>>>>>>>> alternative would be option 3, but a single extension name covering
>>>>>>>> two storage layouts ties the layout to a JSON metadata field rather
>>>>>>>> than to the type name itself, which is easier for downstream tooling
>>>>>>>> to get wrong I believe. What do you think?
>>>>>>> I agree that option 2 sounds best, the tensor analogy is spot-on.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Antoine.
>>>>>>>

Re: [DISCUSS] Add arrow.range canonical extension type for bounded ranges

Posted by gi...@frhoelzlwimmer.de.

Hi Felipe,

 > I feel like you should have different types depending on whether a
 > bound is open or not.

I found your idea tempting and was convinced by it for a while. But
after looking into how Arrow C++ actually resolves kernels, I now lean
the other way.

 > Routing compute functions based on metadata would be hellish and would
 > force compute function writers to care about all possible values of
 > the metadata when a new function on ranges is introduced.

I checked this, and separate types help less than it seems. All
extension types report `Type::EXTENSION`, so the kernel matcher cannot
distinguish `half_open_range` from `closed_range` on its own. You still
need a custom matcher on the extension name. That is the same work as
matching one `arrow.range` and reading its `closed` flag.

Also, `closed` is not parsed at compute time. It is read once when the
type is reconstructed. After that it is a typed enum accessor,
`RangeType::closed()`, just like reading a decimal's scale.

Where you are right: If a function author does not account for
closedness at all, separate types fail more safely. Passing a
`closed_range` to a half-open-only kernel would give a loud "no kernel
matching input types", while a single parameterized type can be
silently mishandled.

However, I believe that there is a better fix than having more types:

1. An author handles closedness but forgets a case:

    That is caught at compile time. A switch over the four `RangeClosed`
    values with no default becomes a compile error if a case is missing
    (under -Wswitch/-Werror). The existing `ClosedToString` already
    relies on this.

    Four separate types give less protection here, because forgetting to
    register the kernel for `closed_range` would still compile and only
    fail at runtime.

2. An author never reads `closed` at all:

    In your scheme that author registers `contains()` for
    `half_open_range` only, and a `closed_range` input is rejected with
    "no kernel matching input types".

    My parameter design would give the same protection through the
    kernel's declared input type. A kernel registered with a
    `match::RangeClosed(Left)` matcher is only picked for `arrow.range`
    arrays whose `closed == Left`, and a `closed == Both` input is
    rejected with the same error.
    This is the same way `match::TimestampTypeUnit` works already.

 > With the different types, you could have a canonicalization layer
 > that allocates a new array before a function learns to handle all
 > possible range types.

That layer fits on top of the matcher and does not need separate types.
A `canonicalize_to_half_open(array)` helper can convert any discrete
`arrow.range` to `closed == Left`, and a kernel author can call it
instead of rejecting the input.

However, please note that canonicalizing to half-open does not work for
continuous subtypes, as discussed earlier in this thread. Technically
every Arrow type has a smallest representable increment (e.g. `[1, 5]`
over `double` would become `[1, nextafter(5))`). But such a conversion
would change the bound, the result would differ after a lossless cast
between precisions or timestamp units, and a round trip through
PostgreSQL's `numrange` would keep the converted
`[1, 5.000000000000001)` instead of restoring `[1, 5]`.

This applies to both designs. A half-open-only `contains()` over
`numrange` must handle open and closed bounds directly either way.

 > Making the openness per value will complicate columnar compute
 > functions, even though it simplifies transport from databases like
 > PostgreSQL.

I agree, but per-value closedness is the only lossless option for
continuous ranges. PostgreSQL stores the inclusive/exclusive flags per
value, and one `numrange` column can legitimately hold both (1, 5) and
[1, 5] values. None of the four fixed types can represent such a
mixed-closedness column. So a per-value type has to exist regardless of
what we do with the fixed case.

 > At the same time, expecting application consumers of an ADBC driver to
 > check the open/closed flags per value is a big ask.

Here I disagree. In a `range_inc` column the inclusivity varies per row.
A consumer that cares about inclusivity has to read it per row in any
design, because that is where the information is.

Applications that care about per-value closedness already do this
(e.g. temporal intervals, coordinates). A consumer that does not care
can cast the column to the fixed-closedness `arrow.range` and drop
the flags.

One more thing: Four fixed types are quite unidiomatic for Arrow. They
would have identical storage, `Struct<lower, upper>`, and differ only
in a semantic flag. Arrow normally expresses that as a parameter on one
type rather than a separate type per value (`decimal` carries scale and
precision, `timestamp` carries unit and time zone). It would also mean
five canonical types to specify and maintain instead of two, across
every implementation: C++, Rust, Go, Java, JS, Swift, nanoarrow, and so
on.

Bottom line, I would prefer keeping one `arrow.range` type with a
`closed` parameter (the same shape as pandas' IntervalArray) plus one
per-value type. What do you think?

Best,
Hoeze

PS: What are the next steps once this discussion has converged?

On 11.09.26 02:28, git.ich@frhoelzlwimmer.de wrote:
> Dear Rok, dear Felipe,
>
> yes, I'm definitely interested in getting it over the finish line :)
>
> @Felipe I'm very sorry, somehow your message slipped my attention.
> Let me sleep over it, and I'll give you a founded response tomorrow ^^
>
> Best,
> Florian
>
>
> On 10.09.26 18:49, Rok Mihevc wrote:
>> Hey all,
>>
>> Is there still interest in standardizing this? I thought we were on a good
>> track and it'd be nice to complete it.
>>
>> Best,
>> Rok
>>
>>
>>
>> On Mon, Jun 8, 2026 at 7:26 PM Felipe Oliveira Carvalho <fe...@gmail.com>
>> wrote:
>>
>>> I feel like you should have different types depending on whether a bound is
>>> open or not.
>>>
>>> [.., ..) arrow.half_open_range
>>> (.., ..) arrow.closed_range
>>> (.., ..] arrow.lower_half_open_range
>>> [.., ..] arrow.open_range
>>>
>>> Routing compute functions based on metadata would be hellish and would
>>> force compute function writers to care about all possible values of the
>>> metadata when a new function on ranges is introduced. With the different
>>> types, you could have a canonicalization layer that allocates a new array
>>> before a function learns to handle all possible range types.
>>>
>>> Example: a contains() implementation that computes directly only
>>> arrow.half_open_range arrays, but when that is not the input, it calls a
>>> utility that can canonicalize every range type to arrow.half_open_range.
>>>
>>> Making the openness per value will complicate columnar compute functions,
>>> even though it simplifies transport from databases like PostgreSQL. You
>>> would not need to canonicalize the output from a PostgreSQL query into one
>>> of the range types in the ADBC driver, for instance. At the same time,
>>> expecting application consumers of an ADBC driver to check the open/closed
>>> flags per value is a big ask.
>>>
>>> -- 
>>> Felipe
>>>
>>> On Sun, Jun 7, 2026 at 11:23 AM Hoeze <gi...@frhoelzlwimmer.de> wrote:
>>>
>>>> Dear all,
>>>>
>>>> What is your opinion on naming the range types with a fixed closedness
>>> and
>>>> a per-value closedness?
>>>> Currently, they are called `RangeType` and `RangeIncType`, which Rok
>>>> considers poor naming (and I agree).
>>>>
>>>> Some ideas:
>>>>
>>>>   1. RangeType | VarRangeType
>>>>   2. RangeType | PerValueRangeType
>>>>   3. RangeType | GranularRangeType
>>>>   4. FixedClosednessRangeType | VariableClosednessRangeType
>>>>
>>>> My personal favorite is (4), as it mirrors the tensor type naming and
>>>> points out the difference.
>>>> However, it sounds quite long and clumpy.
>>>>
>>>> What do you think?
>>>>
>>>> Best,
>>>> Hoeze
>>>>
>>>> On 04/06/2026 22:30, Hoeze wrote:
>>>>> Thank you for your feedback, Antoine.
>>>>>
>>>>> I updated my draft PR and added the 'range_inc' type:
>>>>> https://github.com/apache/arrow/pull/50028/
>>>>>
>>>>> Please let me know if you have any further suggestions :)
>>>>>
>>>>> Best,
>>>>> Hoeze
>>>>>
>>>>> Am 02.06.26 um 19:00 schrieb Antoine Pitrou:
>>>>>> Le 25/05/2026 à 16:54, Hoeze a écrit :
>>>>>>> Yes, you're right, the current proposal would probably not be
>>>>>>> sufficient for continuous PostgreSQL ranges.
>>>>>>>
>>>>>>> Column level boundary flags were intentional as it allows to check
>>>>>>> closedness in the schema instead of during runtime. This is also how
>>>>>>> Pandas' `IntervalArray`/`IntervalIndex` works.
>>>>>>>
>>>>>>> PostgreSQL's built-in discrete ranges (`int4range`, `int8range`,
>>>>>>> `daterange`) canonicalize to left-closed intervals; here my proposal
>>>>>>> would be sufficient. However, continuous ranges (`numrange`,
>>>>>>> `tsrange`, `tstzrange`, ...) cannot be canonicalized. In this case
>>>>>>> my proposal would indeed not be flexible enough.
>>>>>>>
>>>>>>> I could imagine a number of possible solutions to this shortcoming:
>>>>>>>
>>>>>>>      * Union type of all four closedness versions:
>>>>>>>        Possible but not very elegant. Would shift the implementation
>>>>>>>        burden towards the applications, that have to support union
>>>>>>>        types.
>>>>>>>
>>>>>>>      * Create a separate canonical data type for per-value boundary
>>>>>>>        flags:
>>>>>>>        Storage type `Struct<lower: T, upper: T, lower_inc: bool,
>>>>>>>        upper_inc: bool>`, mirroring PostgreSQL's internal
>>>>>>>        representation. Both types would coexist: `arrow.range` for the
>>>>>>>        uniform case (and for canonicalized discrete PostgreSQL
>>> ranges),
>>>>>>>        and e.g. `arrow.range_inc` for continuous (PostgreSQL) ranges.
>>>>>>>
>>>>>>>      * Extend `arrow.range` itself with a per-value mode:
>>>>>>>        Keep a single extension type, but allow
>>>>>>>        `{"closed": "per_value"}` in the metadata, in which case the
>>>>>>>        storage struct gains two boolean fields `lower_inc` and
>>>>>>>        `upper_inc`. One extension name, two storage layouts. Simpler
>>>>>>>        from a type-registry standpoint, slightly more conditional
>>> logic
>>>>>>>        in implementations.
>>>>>>>
>>>>>>>      * Always store per-value flags:
>>>>>>>        Drop the metadata key entirely and always use
>>>>>>>        `Struct<lower: T, upper: T, lower_inc: bool, upper_inc: bool>`.
>>>>>>>        Two extra bytes per row uncompressed, but highly
>>> RLE/dictionary-
>>>>>>>        friendly when uniform (which it usually is). Maximally simple
>>> to
>>>>>>>        specify, at the cost of some overhead in the common
>>> pandas-style
>>>>>>>        case.
>>>>>>>
>>>>>>> I currently lean towards the second option, as it preserves the
>>>>>>> schema-level check for the common case while still giving continuous,
>>>>>>> per-value closedness ranges a lossless path. Fixed-shape tensor vs.
>>>>>>> variable-shape tensor extension types went the same route. The main
>>>>>>> alternative would be option 3, but a single extension name covering
>>>>>>> two storage layouts ties the layout to a JSON metadata field rather
>>>>>>> than to the type name itself, which is easier for downstream tooling
>>>>>>> to get wrong I believe. What do you think?
>>>>>> I agree that option 2 sounds best, the tensor analogy is spot-on.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Antoine.
>>>>>>

Re: [DISCUSS] Add arrow.range canonical extension type for bounded ranges

Posted by gi...@frhoelzlwimmer.de.

Dear Rok, dear Felipe,

yes, I'm definitely interested in getting it over the finish line :)

@Felipe I'm very sorry, somehow your message slipped my attention.
Let me sleep over it, and I'll give you a founded response tomorrow ^^

Best,
Florian


On 10.09.26 18:49, Rok Mihevc wrote:
> Hey all,
>
> Is there still interest in standardizing this? I thought we were on a good
> track and it'd be nice to complete it.
>
> Best,
> Rok
>
>
>
> On Mon, Jun 8, 2026 at 7:26 PM Felipe Oliveira Carvalho <fe...@gmail.com>
> wrote:
>
>> I feel like you should have different types depending on whether a bound is
>> open or not.
>>
>> [.., ..) arrow.half_open_range
>> (.., ..) arrow.closed_range
>> (.., ..] arrow.lower_half_open_range
>> [.., ..] arrow.open_range
>>
>> Routing compute functions based on metadata would be hellish and would
>> force compute function writers to care about all possible values of the
>> metadata when a new function on ranges is introduced. With the different
>> types, you could have a canonicalization layer that allocates a new array
>> before a function learns to handle all possible range types.
>>
>> Example: a contains() implementation that computes directly only
>> arrow.half_open_range arrays, but when that is not the input, it calls a
>> utility that can canonicalize every range type to arrow.half_open_range.
>>
>> Making the openness per value will complicate columnar compute functions,
>> even though it simplifies transport from databases like PostgreSQL. You
>> would not need to canonicalize the output from a PostgreSQL query into one
>> of the range types in the ADBC driver, for instance. At the same time,
>> expecting application consumers of an ADBC driver to check the open/closed
>> flags per value is a big ask.
>>
>> --
>> Felipe
>>
>> On Sun, Jun 7, 2026 at 11:23 AM Hoeze <gi...@frhoelzlwimmer.de> wrote:
>>
>>> Dear all,
>>>
>>> What is your opinion on naming the range types with a fixed closedness
>> and
>>> a per-value closedness?
>>> Currently, they are called `RangeType` and `RangeIncType`, which Rok
>>> considers poor naming (and I agree).
>>>
>>> Some ideas:
>>>
>>>   1. RangeType | VarRangeType
>>>   2. RangeType | PerValueRangeType
>>>   3. RangeType | GranularRangeType
>>>   4. FixedClosednessRangeType | VariableClosednessRangeType
>>>
>>> My personal favorite is (4), as it mirrors the tensor type naming and
>>> points out the difference.
>>> However, it sounds quite long and clumpy.
>>>
>>> What do you think?
>>>
>>> Best,
>>> Hoeze
>>>
>>> On 04/06/2026 22:30, Hoeze wrote:
>>>> Thank you for your feedback, Antoine.
>>>>
>>>> I updated my draft PR and added the 'range_inc' type:
>>>> https://github.com/apache/arrow/pull/50028/
>>>>
>>>> Please let me know if you have any further suggestions :)
>>>>
>>>> Best,
>>>> Hoeze
>>>>
>>>> Am 02.06.26 um 19:00 schrieb Antoine Pitrou:
>>>>> Le 25/05/2026 à 16:54, Hoeze a écrit :
>>>>>> Yes, you're right, the current proposal would probably not be
>>>>>> sufficient for continuous PostgreSQL ranges.
>>>>>>
>>>>>> Column level boundary flags were intentional as it allows to check
>>>>>> closedness in the schema instead of during runtime. This is also how
>>>>>> Pandas' `IntervalArray`/`IntervalIndex` works.
>>>>>>
>>>>>> PostgreSQL's built-in discrete ranges (`int4range`, `int8range`,
>>>>>> `daterange`) canonicalize to left-closed intervals; here my proposal
>>>>>> would be sufficient. However, continuous ranges (`numrange`,
>>>>>> `tsrange`, `tstzrange`, ...) cannot be canonicalized. In this case
>>>>>> my proposal would indeed not be flexible enough.
>>>>>>
>>>>>> I could imagine a number of possible solutions to this shortcoming:
>>>>>>
>>>>>>      * Union type of all four closedness versions:
>>>>>>        Possible but not very elegant. Would shift the implementation
>>>>>>        burden towards the applications, that have to support union
>>>>>>        types.
>>>>>>
>>>>>>      * Create a separate canonical data type for per-value boundary
>>>>>>        flags:
>>>>>>        Storage type `Struct<lower: T, upper: T, lower_inc: bool,
>>>>>>        upper_inc: bool>`, mirroring PostgreSQL's internal
>>>>>>        representation. Both types would coexist: `arrow.range` for the
>>>>>>        uniform case (and for canonicalized discrete PostgreSQL
>> ranges),
>>>>>>        and e.g. `arrow.range_inc` for continuous (PostgreSQL) ranges.
>>>>>>
>>>>>>      * Extend `arrow.range` itself with a per-value mode:
>>>>>>        Keep a single extension type, but allow
>>>>>>        `{"closed": "per_value"}` in the metadata, in which case the
>>>>>>        storage struct gains two boolean fields `lower_inc` and
>>>>>>        `upper_inc`. One extension name, two storage layouts. Simpler
>>>>>>        from a type-registry standpoint, slightly more conditional
>> logic
>>>>>>        in implementations.
>>>>>>
>>>>>>      * Always store per-value flags:
>>>>>>        Drop the metadata key entirely and always use
>>>>>>        `Struct<lower: T, upper: T, lower_inc: bool, upper_inc: bool>`.
>>>>>>        Two extra bytes per row uncompressed, but highly
>> RLE/dictionary-
>>>>>>        friendly when uniform (which it usually is). Maximally simple
>> to
>>>>>>        specify, at the cost of some overhead in the common
>> pandas-style
>>>>>>        case.
>>>>>>
>>>>>> I currently lean towards the second option, as it preserves the
>>>>>> schema-level check for the common case while still giving continuous,
>>>>>> per-value closedness ranges a lossless path. Fixed-shape tensor vs.
>>>>>> variable-shape tensor extension types went the same route. The main
>>>>>> alternative would be option 3, but a single extension name covering
>>>>>> two storage layouts ties the layout to a JSON metadata field rather
>>>>>> than to the type name itself, which is easier for downstream tooling
>>>>>> to get wrong I believe. What do you think?
>>>>> I agree that option 2 sounds best, the tensor analogy is spot-on.
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>

Re: [DISCUSS] Add arrow.range canonical extension type for bounded ranges

Posted by Rok Mihevc <ro...@gmail.com>.

Hey all,

Is there still interest in standardizing this? I thought we were on a good
track and it'd be nice to complete it.

Best,
Rok



On Mon, Jun 8, 2026 at 7:26 PM Felipe Oliveira Carvalho <fe...@gmail.com>
wrote:

> I feel like you should have different types depending on whether a bound is
> open or not.
>
> [.., ..) arrow.half_open_range
> (.., ..) arrow.closed_range
> (.., ..] arrow.lower_half_open_range
> [.., ..] arrow.open_range
>
> Routing compute functions based on metadata would be hellish and would
> force compute function writers to care about all possible values of the
> metadata when a new function on ranges is introduced. With the different
> types, you could have a canonicalization layer that allocates a new array
> before a function learns to handle all possible range types.
>
> Example: a contains() implementation that computes directly only
> arrow.half_open_range arrays, but when that is not the input, it calls a
> utility that can canonicalize every range type to arrow.half_open_range.
>
> Making the openness per value will complicate columnar compute functions,
> even though it simplifies transport from databases like PostgreSQL. You
> would not need to canonicalize the output from a PostgreSQL query into one
> of the range types in the ADBC driver, for instance. At the same time,
> expecting application consumers of an ADBC driver to check the open/closed
> flags per value is a big ask.
>
> --
> Felipe
>
> On Sun, Jun 7, 2026 at 11:23 AM Hoeze <gi...@frhoelzlwimmer.de> wrote:
>
> > Dear all,
> >
> > What is your opinion on naming the range types with a fixed closedness
> and
> > a per-value closedness?
> > Currently, they are called `RangeType` and `RangeIncType`, which Rok
> > considers poor naming (and I agree).
> >
> > Some ideas:
> >
> >  1. RangeType | VarRangeType
> >  2. RangeType | PerValueRangeType
> >  3. RangeType | GranularRangeType
> >  4. FixedClosednessRangeType | VariableClosednessRangeType
> >
> > My personal favorite is (4), as it mirrors the tensor type naming and
> > points out the difference.
> > However, it sounds quite long and clumpy.
> >
> > What do you think?
> >
> > Best,
> > Hoeze
> >
> > On 04/06/2026 22:30, Hoeze wrote:
> > > Thank you for your feedback, Antoine.
> > >
> > > I updated my draft PR and added the 'range_inc' type:
> > > https://github.com/apache/arrow/pull/50028/
> > >
> > > Please let me know if you have any further suggestions :)
> > >
> > > Best,
> > > Hoeze
> > >
> > > Am 02.06.26 um 19:00 schrieb Antoine Pitrou:
> > >>
> > >> Le 25/05/2026 à 16:54, Hoeze a écrit :
> > >>> Yes, you're right, the current proposal would probably not be
> > >>> sufficient for continuous PostgreSQL ranges.
> > >>>
> > >>> Column level boundary flags were intentional as it allows to check
> > >>> closedness in the schema instead of during runtime. This is also how
> > >>> Pandas' `IntervalArray`/`IntervalIndex` works.
> > >>>
> > >>> PostgreSQL's built-in discrete ranges (`int4range`, `int8range`,
> > >>> `daterange`) canonicalize to left-closed intervals; here my proposal
> > >>> would be sufficient. However, continuous ranges (`numrange`,
> > >>> `tsrange`, `tstzrange`, ...) cannot be canonicalized. In this case
> > >>> my proposal would indeed not be flexible enough.
> > >>>
> > >>> I could imagine a number of possible solutions to this shortcoming:
> > >>>
> > >>>     * Union type of all four closedness versions:
> > >>>       Possible but not very elegant. Would shift the implementation
> > >>>       burden towards the applications, that have to support union
> > >>>       types.
> > >>>
> > >>>     * Create a separate canonical data type for per-value boundary
> > >>>       flags:
> > >>>       Storage type `Struct<lower: T, upper: T, lower_inc: bool,
> > >>>       upper_inc: bool>`, mirroring PostgreSQL's internal
> > >>>       representation. Both types would coexist: `arrow.range` for the
> > >>>       uniform case (and for canonicalized discrete PostgreSQL
> ranges),
> > >>>       and e.g. `arrow.range_inc` for continuous (PostgreSQL) ranges.
> > >>>
> > >>>     * Extend `arrow.range` itself with a per-value mode:
> > >>>       Keep a single extension type, but allow
> > >>>       `{"closed": "per_value"}` in the metadata, in which case the
> > >>>       storage struct gains two boolean fields `lower_inc` and
> > >>>       `upper_inc`. One extension name, two storage layouts. Simpler
> > >>>       from a type-registry standpoint, slightly more conditional
> logic
> > >>>       in implementations.
> > >>>
> > >>>     * Always store per-value flags:
> > >>>       Drop the metadata key entirely and always use
> > >>>       `Struct<lower: T, upper: T, lower_inc: bool, upper_inc: bool>`.
> > >>>       Two extra bytes per row uncompressed, but highly
> RLE/dictionary-
> > >>>       friendly when uniform (which it usually is). Maximally simple
> to
> > >>>       specify, at the cost of some overhead in the common
> pandas-style
> > >>>       case.
> > >>>
> > >>> I currently lean towards the second option, as it preserves the
> > >>> schema-level check for the common case while still giving continuous,
> > >>> per-value closedness ranges a lossless path. Fixed-shape tensor vs.
> > >>> variable-shape tensor extension types went the same route. The main
> > >>> alternative would be option 3, but a single extension name covering
> > >>> two storage layouts ties the layout to a JSON metadata field rather
> > >>> than to the type name itself, which is easier for downstream tooling
> > >>> to get wrong I believe. What do you think?
> > >>
> > >> I agree that option 2 sounds best, the tensor analogy is spot-on.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
>