You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/01/31 20:35:30 UTC
[I] Optimize `to_timestamp` [arrow-datafusion]
alamb opened a new issue, #9090:
URL: https://github.com/apache/arrow-datafusion/issues/9090
### Is your feature request related to a problem or challenge?
_No response_
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `to_timestamp` [arrow-datafusion]
Posted by "istvan-fodor (via GitHub)" <gi...@apache.org>.
istvan-fodor commented on issue #9090:
URL: https://github.com/apache/arrow-datafusion/issues/9090#issuecomment-1998676975
@alamb, @Omega359 I took at look at the benchmarks and I had some questions.
It looks to me that the code that is benchmarked doesn't actually create timestamps, it just creates the expressions that **if** evaluated would/should return timestamps. I checked in the debugger and could never catch anything in `ToTimestampFunc::invoke()`. Also, when I printed out the results of the `to_timestamp` calls form the benchmark I saw these:
```
to_timestamp(Utf8("1997-01-31T09:26:56.123Z"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%dT%H:%M:%S%.f%Z"))
to_timestamp(Utf8("1997-01-31T09:26:56.123-05:00"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%dT%H:%M:%S%.f%z"))
to_timestamp(Utf8("1997-01-31 09:26:56.123-05:00"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%d %H:%M:%S%.f%Z"))
to_timestamp(Utf8("2023-01-01 04:05:06.789 -08"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%d %H:%M:%S%.f %#z"))
to_timestamp(Utf8("1997-01-31T09:26:56.123"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%dT%H:%M:%S%.f"))
to_timestamp(Utf8("1997-01-31 09:26:56.123"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%d %H:%M:%S%.f"))
to_timestamp(Utf8("1997-01-31 09:26:56"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%d %H:%M:%S"))
to_timestamp(Utf8("1997-01-31 092656"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%d %H%M%S"))
to_timestamp(Utf8("1997-01-31 092656+04:00"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%d %H%M%S%:z"))
to_timestamp(Utf8("Sun Jul 8 00:34:60 2001"), Utf8("%+"), Utf8("%c"), Utf8("%Y-%m-%d 00:00:00"))
```
Am I reading this right or am I missing this completely? Is the goal here to speed up str -> timestamp creation or to speed up the creation of the expression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `to_timestamp` [arrow-datafusion]
Posted by "manoj-inukolunu (via GitHub)" <gi...@apache.org>.
manoj-inukolunu commented on issue #9090:
URL: https://github.com/apache/arrow-datafusion/issues/9090#issuecomment-1922813395
@alamb can i give this a shot too please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `to_timestamp` [arrow-datafusion]
Posted by "vojtechtoman (via GitHub)" <gi...@apache.org>.
vojtechtoman commented on issue #9090:
URL: https://github.com/apache/arrow-datafusion/issues/9090#issuecomment-2007170901
https://github.com/apache/arrow-datafusion/pull/9694 is still an early draft but I was already able to improve the performance of `to_timestamp` (without format for now) by about 15% percent on my machine, and I think there is still quite some potential there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `to_timestamp` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9090:
URL: https://github.com/apache/arrow-datafusion/issues/9090#issuecomment-1998687443
> Am I reading this right or am I missing this completely? Is the goal here to speed up str -> timestamp creation or to speed up the creation of the expression?
I believe @Omega359 fixed this in https://github.com/apache/arrow-datafusion/pull/9608 (which merged a few hours ago)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `to_timestamp` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #9090: Optimize `to_timestamp`
URL: https://github.com/apache/datafusion/issues/9090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize `to_timestamp` [arrow-datafusion]
Posted by "vojtechtoman (via GitHub)" <gi...@apache.org>.
vojtechtoman commented on issue #9090:
URL: https://github.com/apache/arrow-datafusion/issues/9090#issuecomment-2024884187
While #9694 is still work in progress, I managed to speed up some cases of `to_timestamp(timestamp, format) in https://github.com/apache/arrow-datafusion/pull/9833.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `to_timestamp` [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9090:
URL: https://github.com/apache/arrow-datafusion/issues/9090#issuecomment-2025308331
Awesome -- thank you @vojtechtoman -- I'll check it out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org