You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "icexelloss (via GitHub)" <gi...@apache.org> on 2023/02/15 21:22:04 UTC
[GitHub] [arrow] icexelloss opened a new issue, #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
icexelloss opened a new issue, #34210:
URL: https://github.com/apache/arrow/issues/34210
### Describe the bug, including details regarding any error messages, version, and platform.
I am testing performance of casting datatypes with pyarrow Table and saw some unexpected performance.
In short, it seems that casting a column from "tz-naive" to "tz-utc" is much slower than casting from "tz-naive" to "int64", which is unexpected because I think both of these should be metadata-only change.
Here is a partial repo:
```
In [5]: df = pd.DataFrame({'time': np.arange(100 * 1000 * 1000)})
In [6]: table = pa.Table.from_pandas(df)
In [8]: schema_naive = pa.schema([pa.field('time' , pa.timestamp('ns'))])
In [9]: schema_tz = pa.schema([pa.field('time' , pa.timestamp('ns', tz='UTC'))])
In [10]: table = table.cast(schema_naive)
In [14]: schema_int = pa.schema([pa.field('time' , pa.int64()))])
In [16]: %time table.cast(schema_int)
CPU times: user 114 µs, sys: 30 µs, total: 144 µs
Wall time: 231 µs
Out[16]:
pyarrow.Table
time: int64
----
time: [[0,1,2,3,4,...,99999995,99999996,99999997,99999998,99999999]]
In [17]: %time table.cast(schema_tz)
CPU times: user 119 ms, sys: 140 ms, total: 260 ms
Wall time: 259 ms
Out[17]:
pyarrow.Table
time: timestamp[ns, tz=UTC]
----
time: [[1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000001,1970-01-01 00:00:00.000000002,1970-01-01 00:00:00.000000003,1970-01-01 00:00:00.000000004,...,1970-01-01 00:00:00.099999995,1970-01-01 00:00:00.099999996,1970-01-01 00:00:00.099999997,1970-01-01 00:00:00.099999998,1970-01-01 00:00:00.099999999]]
In [18]: table
Out[18]:
pyarrow.Table
time: timestamp[ns]
----
time: [[1970-01-01 00:00:00.000000000,1970-01-01 00:00:00.000000001,1970-01-01 00:00:00.000000002,1970-01-01 00:00:00.000000003,1970-01-01 00:00:00.000000004,...,1970-01-01 00:00:00.099999995,1970-01-01 00:00:00.099999996,1970-01-01 00:00:00.099999997,1970-01-01 00:00:00.099999998,1970-01-01 00:00:00.099999999]]
In [19]: pa.__version__
Out[19]: '11.0.0'
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rok commented on issue #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #34210:
URL: https://github.com/apache/arrow/issues/34210#issuecomment-1433583552
This would would definitely be nice to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] icexelloss commented on issue #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
Posted by "icexelloss (via GitHub)" <gi...@apache.org>.
icexelloss commented on issue #34210:
URL: https://github.com/apache/arrow/issues/34210#issuecomment-1432103431
FWIW I found this comment about the behavior: https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/scalar_cast_temporal.cc#L156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] icexelloss commented on issue #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
Posted by "icexelloss (via GitHub)" <gi...@apache.org>.
icexelloss commented on issue #34210:
URL: https://github.com/apache/arrow/issues/34210#issuecomment-1433627836
@rok This might be a silly question but why don't we dynamically dispatch to a zero copy / no op functions here if the units are the same?
https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/scalar_cast_temporal.cc#L158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on issue #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34210:
URL: https://github.com/apache/arrow/issues/34210#issuecomment-1432727111
Indeed, that seems to be due to not doing this a zero-copy cast because the data types are "different" (while only the unit matters, the timezone can be ignored for this operation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rok commented on issue #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #34210:
URL: https://github.com/apache/arrow/issues/34210#issuecomment-1433637359
@icexelloss yeah, no op seems like the thing to do. I wonder if zero copy is possible or does the fact we're working with batches prevent that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34210:
URL: https://github.com/apache/arrow/issues/34210#issuecomment-1433982171
I don't think batches should be a problem. It seems we only have the comment from @wesm to go on here. Given that it was made during a rather large refactor my guess is this is more of a "todo" and less of a "this is a concern". I think it should be pretty safe to zero copy (and it is concerning that we don't).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rok commented on issue #34210: [Python] Unexpected performance when casting from tz-naive to tz-aware timestamps
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #34210:
URL: https://github.com/apache/arrow/issues/34210#issuecomment-1436804172
Ok, I'll give it a shot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rok closed issue #34210: [C++] Make casting timestamp and duration zero-copy when TimeUnit matches
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok closed issue #34210: [C++] Make casting timestamp and duration zero-copy when TimeUnit matches
URL: https://github.com/apache/arrow/issues/34210
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org