You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "mroeschke (via GitHub)" <gi...@apache.org> on 2023/07/14 00:34:37 UTC

[GitHub] [arrow] mroeschke opened a new issue, #36677: [Python]: Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

mroeschke opened a new issue, #36677:
URL: https://github.com/apache/arrow/issues/36677

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   ```
   >>> import pyarrow as pa
   >>> pa.__version__
   '12.0.1'
   >>> pa.scalar(43260.0).cast(pa.time32("s"))
   <pyarrow.Time32Scalar: datetime.time(12, 1)>
   
   >>> pa.__version__
   '13.0.0.dev497'
   >>> pa.scalar(43260.0).cast(pa.time32("s"))
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "pyarrow/scalar.pxi", line 91, in pyarrow.lib.Scalar.cast
     File "/opt/miniconda3/envs/pyarrow-pandas-dev/lib/python3.11/site-packages/pyarrow/compute.py", line 403, in cast
       return call_function("cast", [arr], options, memory_pool)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "pyarrow/_compute.pyx", line 572, in pyarrow._compute.call_function
     File "pyarrow/_compute.pyx", line 367, in pyarrow._compute.Function.call
     File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
   pyarrow.lib.ArrowNotImplementedError: Unsupported cast from double to time32 using function cast_time32
   ```
   
   Not sure if this will be an intentional change, but it appears casting from double to temporal types has changed. In theory for the example above `43260.0` can be cast exactly back to `datetime.time(12, 1)`. I could see how for a value like `43260.1` there would be implicit rounding in pyarrow 12
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mroeschke commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1642504781

   With respect to the original issue, happy to close it if unsafe casting from temporal to numeric is intended in pyarrow 13


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mroeschke commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1636166643

   Thanks for pointing that out @rohanjain101! Moreover, it appears that cast gained a new safe keyword but it doesn't handle numeric to temporal casting https://github.com/apache/arrow/pull/35395#issuecomment-1636165320 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mroeschke commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1642424789

   In pandas, the use case is to perform reduction functions (e.g. mean) on temporal values, and since the pyarrow reduction kernels don't support temporal values, we cast them numeric first (assuming they are some x value relative to y) and then back.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1636226523

   @mroeschke does it work to go from double -> int64 -> temporal?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1636227041

   (or I guess int32 in this case)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev [arrow]

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke closed issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev
URL: https://github.com/apache/arrow/issues/36677


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1642398959

   It it common to store temporal values as floating point values?  My initial reaction is that this is unusual and requiring an intermediate cast is acceptable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev [arrow]

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1804149288

   Seems like this is intended behavior so closing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mroeschke commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1636272601

   > @mroeschke does it work to go from double -> int64 -> temporal?
   
   Yeah this case seems to work
   
   ```
   >>> import pyarrow as pa
   >>> pa.scalar(1.1).cast(pa.int32(), safe=False).cast(pa.time32("s"), safe=False)
   <pyarrow.Time32Scalar: datetime.time(0, 0, 1)>
   >>> pa.__version__
   '13.0.0.dev497'
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1642495160

   So this means that pyarrow 13.0.0 will break `Series[timestamp].mean()` (with the pyarrow backend)?
   
   That's annoying .., but in general I also agree that the extra cast to int is fine (as we required for arrays already before as well). For example, in pandas we also don't allow casting timestamp to float (but do the other way around, a bit inconsistent).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] mroeschke commented on issue #36677: [Python] Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1642502695

   Yeah pandas caught that breaking in the nightly arrow job, but I think I addressed `Series[timestamp<pyarrow>].mean()` by casting to int in pandas https://github.com/pandas-dev/pandas/pull/54110/files


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] rohanjain101 commented on issue #36677: [Python]: Casting round double scalars to temporal types raising ArrowNotImplementedError on pyarrow 13 dev

Posted by "rohanjain101 (via GitHub)" <gi...@apache.org>.
rohanjain101 commented on issue #36677:
URL: https://github.com/apache/arrow/issues/36677#issuecomment-1635179229

   https://github.com/apache/arrow/pull/35395
   
   I think it's due to this PR where scalars were using a different cast method instead of the compute kernel.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org