You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/10 09:29:28 UTC
[GitHub] [arrow-datafusion] ike560 opened a new issue, #3922: Internal error in CAST from Timestamp[us]
ike560 opened a new issue, #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922
**Describe the bug**
The following error occurs in cast of timestamp.
Exception: Internal error: Unsupported CAST from Timestamp(Microsecond, None) to Int32. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
**To Reproduce**
```py
import datafusion
ctx = datafusion.SessionContext()
ctx.register_parquet('nyc_taxi', 'nyc_taxi.parquet')
sql = '''
WITH a AS
(SELECT
tpep_pickup_datetime,
tpep_dropoff_datetime,
CAST(tpep_pickup_datetime AS INTEGER) AS pickup,
CAST(tpep_dropoff_datetime AS INTEGER) AS dropoff
FROM nyc_taxi)
SELECT
CAST(AVG(dropoff - pickup) AS NUMERIC(5, 2))
FROM a
'''
df = ctx.sql(sql)
df.show()
```
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
Input In [5], in <module>
4 sql = '''
5 WITH a AS
6 (SELECT
(...)
14 FROM a
15 '''
16 df = ctx.sql(sql)
---> 17 df.show()
Exception: Internal error: Unsupported CAST from Timestamp(Microsecond, None) to Int32. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker
**Additional context**
Latest version : datafusion 0.6.0 (python)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
waitingkuo commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1287399593
@alamb i think int32 isn't large enough for microsecond scale.
`cast`ing timestamp[us] to integer returns number of microseconds from 1970, but 2^32-1 microseonds is just too small.
```bash
❯ select to_timestamp_micros(4294967295);
+--------------------------------------+
| totimestampmicros(Int64(4294967295)) |
+--------------------------------------+
| 1970-01-01 01:11:34.967295 |
+--------------------------------------+
1 row in set. Query took 0.000 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ike560 commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
ike560 commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1287636492
Thank you all.
I understand that CAST does not support timestamp to INT32/INT64.
The internal error message is misleading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ike560 closed issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
ike560 closed issue #3922: Internal error in CAST from Timestamp[us]
URL: https://github.com/apache/arrow-datafusion/issues/3922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1287360445
The root cause is that the arrow cast kernel doesn't support converting from timestamp --> int32 (as @waitingkuo mentions)
To fix this we çould extend the support support in https://github.com/apache/arrow-rs/blob/c7f7606/arrow/src/compute/kernels/cast.rs#L253-L254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
waitingkuo commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1289094859
@ike560 Cast does support timestamp to INT64, you can try `CAST(tpep_pickup_datetime AS BIGINT)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1287416752
So maybe a better error message is the best solution here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ike560 commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
ike560 commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1289984249
Thanks @waitingkuo
I tried it, CAST to BIGINT was OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ike560 commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
ike560 commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1310003523
Close
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1287356567
The fact this is reported as an internal error is not ideal. I will change the error type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] waitingkuo commented on issue #3922: Internal error in CAST from Timestamp[us]
Posted by GitBox <gi...@apache.org>.
waitingkuo commented on issue #3922:
URL: https://github.com/apache/arrow-datafusion/issues/3922#issuecomment-1287006273
there's no implementaton for casting timestamp to int32 now. you can cast it to int64 for now (bigint)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #3922: Internal error in CAST from Timestamp[us]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #3922: Internal error in CAST from Timestamp[us]
URL: https://github.com/apache/arrow-datafusion/issues/3922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org