You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "randolf-scholz (via GitHub)" <gi...@apache.org> on 2025/10/30 15:39:08 UTC
[I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
randolf-scholz opened a new issue, #48003:
URL: https://github.com/apache/arrow/issues/48003
### Describe the enhancement requested
`numpy` supports multiplying/dividing `timedelta` values with floats and performs automatic rounding to the current unit, e.g.
```python
>>> import numpy as np
>>> x = np.array([0, 1, 2, 3, 4, 5], dtype="timedelta64[s]")
>>> x * 0.5
array([0, 0, 1, 1, 2, 2], dtype='timedelta64[s]')
>>> x * 1.5
array([0, 1, 3, 4, 6, 7], dtype='timedelta64[s]')
```
This is very useful for ML applications where one needs to transform timestamps first to durations and then to floats (and back after performing a forecast). It would be very nice if `pyarrow` supported this as well, so one can directly work with arrow dtypes.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3489764697
I haven't worked on the compute module so I cannot really comment.
From PyArrow view, though, it does sound like a reasonable kernel to add.
> I think we wanted to map this at some point?
Yes: https://github.com/apache/arrow/issues/32007.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542646617
Well, yes, I think we should have a `multiply` kernel for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542179292
Could you be a bit more specific? :)
The question is if we want kernels that can multiply duration with double and return duration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542611337
Right, I was missing a cast or two, here's how I'd express the multiply kernel with pyarrow:
```python
import pyarrow as pa
from pyarrow import compute as pc
unit = "ms"
a = pa.array([2], pa.duration(unit))
b = pa.array([3.7], pa.float64())
c = pc.multiply(a.cast(pa.int64()), b)
result = pc.round_to_multiple(c, 1).cast(pa.int64()).cast(pa.duration(unit))
```
Question is - should we have a kernel that does all of this in c++ or not. And if yes, should it be under `multiply` or be called something else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3469129622
We also have more calendar aware features like temporal subtraction, component extraction and rounding. I suppose it's best you search for temporal in [this docs page](https://arrow.apache.org/docs/cpp/compute.html#temporal-component-extraction) to discover functions that take temporal type as parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542179219
Could you be a bit more specific? :)
The question is if we want kernels that can multiply duration with double and return duration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "randolf-scholz (via GitHub)" <gi...@apache.org>.
randolf-scholz commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3469226967
@rok I am talking about `multiple(duration, double)` and `divide(duration, double)`. These do not exist
```python
import pyarrow as pa
x = pa.array([1, 2, 3], type=pa.duration("s"))
y = pa.array([0.0, 1.5, 2.9], type=pa.float64())
pa.compute.multiply(x, y)
# pyarrow.lib.ArrowNotImplementedError: Function 'multiply' has no kernel matching input types (duration[s], double)
```
Because `duration` is internally an integer, implementing this operation requires automatic rounding as in `numpy`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3503664556
@randolf-scholz I agree we should have this kernel. I'm just curious at this point if temporal rounding is a useful kernel. :)
> > I think we wanted to map this at some point?
>
> Yes: [#32007](https://github.com/apache/arrow/issues/32007).
That's the one, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "randolf-scholz (via GitHub)" <gi...@apache.org>.
randolf-scholz commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542193550
I think they just missed a cast to integer at the start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542187064
Well, this seems a bit recursive unless I'm misreading:
> * `multiply(duration[xs], double)` -> `multiply(duration[xs], double).round_to_multiple(1).cast(duration[xs])`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub) via github" <gi...@arrow.apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-6752369760
@JonSnow1807 it is hard to review all PRs that are being open (generated) at the moment. If you can make a simple and concise PR I'd review it. But if it turns out it's easier to use an agent by myself I will not, sorry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "JonSnow1807 (via GitHub) via github" <gi...@arrow.apache.org>.
JonSnow1807 commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-6753011575
Thank you so much for being willing to take a look at this. I really appreciate it, and I’m sorry for the long comment above. I know you’re getting a lot of PRs right now.
I’ve opened #55460 and tried to keep it as small as I could: 83 lines in 3 files, about half of which is the test. It only handles float64 and follows the behaviour described above. I used an AI assistant for the first draft, but I’ve gone through every line myself and I’m happy to explain or change anything you prefer.
If it still isn’t worth your time, please feel free to close it. I completely understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542283640
> I think they just missed a cast to integer at the start.
I'll let @rok explain what he means, we don't need another misunderstanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3543476288
@randolf-scholz We'd be happy to help guide you with C++!
Yes kernels are templated. See #7341 for original multiplication (PR was merged even though the label says it wasn't) and #12215 for templating adjustment that allowed `multiply(duration, integer)`. There is also the original kernel buildout issue #25028.
For your reference I suggest you look at something newer, e.g.: #44630 (hyperbolic functions).
If you do start working on this please indicate so here just so we don't accidentally have duplicate effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "randolf-scholz (via GitHub)" <gi...@apache.org>.
randolf-scholz commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3469263797
There are also some other kernels that would be useful for duration, like `min` and `max`, see my issue from ~ 2 years ago: #39233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3469111196
Thanks for reporting @randolf-scholz !
Arrow/PyArrow should support multiplication, division, subtraction and addition via [compute functions](https://arrow.apache.org/docs/cpp/compute.html#arithmetic-functions) which are not mapped to the usual arithmetic symbols yet, making this a bit harder to find. (@raulcd @AlenkaF I think we wanted to map this at some point?)
So in pyarrow you can run:
```python
import pyarrow as pa
import pyarrow.compute as pc
import numpy as np
x = np.array([0, 1, 2, 3, 4, 5], dtype="timedelta64[s]")
y = pa.array(x)
pc.divide(y, 3)
<pyarrow.lib.DurationArray object at 0x76c84b059960>
[
0,
0,
0,
1,
1,
1
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "randolf-scholz (via GitHub)" <gi...@apache.org>.
randolf-scholz commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3476447161
Well, of course one can work around missing kernels by emulating them. But this is quite annoying to do, especially when working with libraries that use `pyarrow` as a backend like `polars` or `pandas`. For instance, existing code that just uses regular arithmetic operators `floats = (time -time.iloc[0]) / unit; floats * unit` will break when attempting to use `pyarrow` backend, as these engines try to look up a nonexisting kernel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3518146007
To circle back:
> [@rok](https://github.com/rok) I am talking about `multiply(duration, double)` and `divide(duration, double)`. These kernels do not exist (tested with version 22.0.0)
We have a request to add two kernels that could be expressed as composites of existing kernels:
* `multiply(duration[xs], double)` -> `multiply(duration[xs], double).round_to_multiple(1).cast(duration[xs])`
* `divide(duration[xs], double)` -> `divide(duration[xs], double).round_to_multiple(1).cast(duration[xs])`
This would be straightforward to do except for maybe the rounding parameters. However I am not sure we want to introduce composite kernels. Perhaps we could have alias kernels (either in c++ or python) that would run the series of kernels that would express the desired computation?
cc @pitrou @jorisvandenbossche what's you opinion on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "randolf-scholz (via GitHub)" <gi...@apache.org>.
randolf-scholz commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542991965
With a dedicated kernel, one should also consider some edge cases and decide whether the computation should align with `numpy`'s precedence:
- `numpy` seems to round towards zero for this operation (e.g. `-1.8 -> -1`), whereas @rok 's suggestion rounds ties towards the next even value.
- How should out-of-bounds values be handled? `numpy` converts these to the special `NaT` value, which is like `NaN` for durations and timestamps, but semantically different from `pyarrow`'s `null` (missing values), although these do seem to get converted to null when importing such an array from numpy:
```python
import numpy as np
import pyarrow as pa
x = np.array([0, +10, +100], dtype="timedelta64[s]")
y = x * 10e16
print(y) # [0, 10e18, 'NaT']
print(pa.array(y)) # [0, 10e18, null]
```
Trying to perform this computation directly in `pyarrow` using @rok 's suggestion results in `ArrowInvalid: Float value 10000000000000000000.000000 was truncated converting to int64`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3473306119
Thanks for compiling the list in #39233, it is useful to see!
I suppose the issue you're describing occurs because multiplying double with `duration[xs]` would need to return a `duration[xs]`, which would requires an implicit rounding step (`duration[xs]` is physically an `int64`). We can discuss adding this as implicit logic, but we try to avoid implicit steps like this. You can work around this by adding a round step:
```python
import pyarrow as pa
deltas = pa.array([1, 2, 3], type=pa.duration("s"))
unit = pa.scalar(1, type=pa.duration("s"))
# forward transform to float:
x = pa.compute.divide(deltas, unit) # gives double
# added rounding step
x = pc.compute.round(x, unit.value).cast("int64") # This won't work if unit is not an integer
# backward transform to duration
y = pa.compute.multiply(x, unit)
```
But are you doing temporal rounding?
```math
\frac{timestamp - reference}{unit} * unit + reference
```
If so you can try:
```python
import pyarrow as pa
timestamps = pa.array([1, 2, 3], type=pa.timestamp("s"))
durations = pa.array([1, 2, 3], type=pa.duration("s"))
y = pa.compute.round_temporal(timestamps, unit="second", multiple=1) # We should have this work on durations too
```
Or go unitless:
```python
import pyarrow as pa
durations = pa.array([1, 2, 3], type=pa.duration("s"))
y = pa.compute.round_to_multiple(durations.cast("int64"), 1)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "randolf-scholz (via GitHub)" <gi...@apache.org>.
randolf-scholz commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3543038961
@rok I have very little C++ experience, but if you give me a reference PR[^1], I may be able to take a look at it during some weekend or new year's holidays.
[^1]: I didn't find #39358 from my earlier issue too enlightening - I guess it's using templated functions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3541911262
> * `multiply(duration[xs], double)` -> `multiply(duration[xs], double).round_to_multiple(1).cast(duration[xs])`
> * `divide(duration[xs], double)` -> `divide(duration[xs], double).round_to_multiple(1).cast(duration[xs])`
How does this make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-3542990901
Perfect, let's do it then. @randolf-scholz would you be interested in contributing a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Add multiply/divide kernels for duration and float/double [arrow]
Posted by "JonSnow1807 (via GitHub) via github" <gi...@arrow.apache.org>.
JonSnow1807 commented on issue #48003:
URL: https://github.com/apache/arrow/issues/48003#issuecomment-6751889563
Proposal before a PR: add `multiply(duration, float64)`, `multiply(float64, duration)` and `divide(duration, float64)`, plus the `_checked` variants, returning a duration in the same unit. No `float / duration`. Results round to nearest with ties to even (5 s × 1.5 = 8 s), like Python’s timedelta, rok’s `round_to_multiple` sketch and Arrow’s float-to-decimal cast. It also keeps the issue’s round trip exact: for 1 to 100000 ns, truncation loses one unit up to 6496 times, rounding loses none. NaN, infinite or out-of-range results are an error in both variants, since converting them to int64 is undefined behaviour (numpy gives NaT on x86 but INT64_MAX on ARM). Division by zero errors like `divide(duration, int64)`. Nulls give null. float32/float16 are cast to float64. The tradeoff: numpy, pandas and polars truncate instead.
Does rounding ties to even (rather than numpy’s truncation) and erroring rather than returning null in the non-checked variants sound right? I have an implementation ready if so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org