You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "0x26res (via GitHub)" <gi...@apache.org> on 2025/04/14 17:39:26 UTC
[I] Support pyarrow.Table with CastOptions [arrow]
0x26res opened a new issue, #46138:
URL: https://github.com/apache/arrow/issues/46138
### Describe the enhancement requested
According to the documentation it should be possible to pass an `option` argument of type `payrrow.compute.CastOption` to `pyarrow.Table.cast`.
But as far as I can tell:
- I can't figure out how to get it to work
- there's no example in the documentation
- there's no test for it (in `python/pyarrow/tests/test_table.py`)
```python
import pandas as pd
import pyarrow as pa
import pyarrow.compute as pc
import pytest
original_schema = pa.schema([pa.field("timestamp", pa.timestamp("ns", "UTC"))])
table = pa.table(
[[pd.to_datetime("2023-03-15T15:15:00.123456789Z")]], schema=original_schema
)
new_schema = pa.schema([pa.field("timestamp", pa.timestamp("us", "UTC"))])
with pytest.raises(
pa.ArrowInvalid,
match=r"Casting from timestamp\[ns, tz=UTC\] to timestamp\[us, tz=UTC\] would lose data: 1678893300123456789",
):
table.cast(new_schema)
with pytest.raises(
ValueError,
match=r"Must either pass values for 'target_type' and 'safe' or pass a value for 'options'",
):
table.cast(new_schema, options=pc.CastOptions(allow_time_truncate=True))
with pytest.raises(
TypeError,
match=r"Argument 'target_type' has incorrect type \(expected pyarrow.lib.DataType, got pyarrow.lib.Schema\)",
):
table.cast(
new_schema,
options=pc.CastOptions(target_type=new_schema, allow_time_truncate=True),
)
with pytest.raises(
ValueError,
match=r"Must either pass values for 'target_type' and 'safe' or pass a value for 'options'",
):
table.cast(
new_schema,
safe=False,
options=pc.CastOptions(target_type=pa.struct(new_schema), allow_time_truncate=True),
)
```
I'm happy to send an PR, but not sure where the problem is exactly.
PS: as a side note, CastOptions can be converted to bool and be interpreted by mistake as the `safe` argument:
```
assert bool(pc.CastOptions(allow_time_truncate=True)) == True
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support pyarrow.Table.cast with CastOptions [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #46138:
URL: https://github.com/apache/arrow/issues/46138#issuecomment-2804288078
I think there is an issue with how `CastOptions` interact between table/compute on the `cast` function. I don't have much experience with how this should work and can't spend much to investigate internally, it has to do with how Table/Column(aka RecordBatch)/compute interact with the cast function, basically table uses:
https://github.com/apache/arrow/blob/b603ea5b27092dc870701baade7cdd50462a3e09/python/pyarrow/table.pxi#L4721-L4725
`field.type` will always have a value and options (on this case) too but then we check that either field and safe or options have to be populated:
https://github.com/apache/arrow/blob/b603ea5b27092dc870701baade7cdd50462a3e09/python/pyarrow/_compute.pyx#L2775-L2779
so from what I can see, for table.cast if we add `CastOptions` this will fail.
If someone can investigate a little more that would be awesome.
As per your specific example you can do (without using `CastOptions` but that's still not solving the issue with the interaction of `CastOptions` on `table.cast()`:
```
>>> table.cast(new_schema, safe=False)
pyarrow.Table
timestamp: timestamp[us, tz=UTC]
----
timestamp: [[2023-03-15 15:15:00.123456Z]]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Support pyarrow.Table.cast with CastOptions [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #46138:
URL: https://github.com/apache/arrow/issues/46138#issuecomment-2804290837
cc @AlenkaF @rok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Support pyarrow.Table.cast with CastOptions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #46138:
URL: https://github.com/apache/arrow/issues/46138#issuecomment-2809340768
I had a look, and I agree there is a bug in the `Table.cast()` and also a case of the error message being unclear, combined with missing documentation and examples.
First of all, this behaviour also occurs when using an array directly:
```python
>>> arr = pa.array([pd.to_datetime("2023-03-15T15:15:00.123456789Z")], pa.timestamp("ns", "UTC"))
>>> arr.cast(pa.timestamp("us", "UTC"))
```
This throws:
```Python
ArrowInvalid: Casting from timestamp[ns, tz=UTC] to timestamp[us, tz=UTC] would lose data: ...
```
When trying to add `CastOptions,` you run into the error from above:
```python
>>> arr.cast(pa.timestamp("us", "UTC"), options=pc.CastOptions(allow_time_truncate=True))
ValueError: Must either pass values for 'target_type' and 'safe' or pass a value for 'options'
```
This behavior is actually intentional — see [this PR comment](https://github.com/apache/arrow/pull/13109#discussion_r904379778). The key is that when using `CastOptions,` the `target_type` should be passed inside the options, not as a separate argument:
```python
>>> arr.cast(options=pc.CastOptions(target_type=pa.timestamp("us", "UTC"), allow_time_truncate=True))
<pyarrow.lib.TimestampArray object at 0x105ad1ba0>
[
2023-03-15 15:15:00.123456Z
]
```
That works correctly and truncates as expected in case of an `Array` or a `ChunkedArray` because, in the mentioned PR, the `target_type` has been made optional. That is not the case for a `Table` or a `RecordBatch` where the `target_schema` is not optional:
https://github.com/apache/arrow/blob/586ed925f4bb4d333f8e2a0beb07564bade355e8/python/pyarrow/table.pxi#L3300
https://github.com/apache/arrow/blob/586ed925f4bb4d333f8e2a0beb07564bade355e8/python/pyarrow/table.pxi#L4666
I think we need to update `cast()` method for `RecordBatch` and `Table`, probably making target schema optional. We would also need to clarify all of the cast functions docstrings and update [the tests in test_compute.py](https://github.com/apache/arrow/blob/586ed925f4bb4d333f8e2a0beb07564bade355e8/python/pyarrow/tests/test_compute.py#L1887-L1902) where we should match exception messages and maybe add some comments and tests for tables and batches.
Happy to help with review if someone wants to send in a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Support pyarrow.Table.cast with CastOptions [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #46138:
URL: https://github.com/apache/arrow/issues/46138#issuecomment-2808513620
I will try to investigate this today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Support pyarrow.Table.cast with CastOptions [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #46138:
URL: https://github.com/apache/arrow/issues/46138#issuecomment-4631453554
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org