You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by "Lee, David (PAG)" <Da...@blackrock.com> on 2023/07/21 16:25:31 UTC
How to adjust pyarrow timestamps using pyarrow.compute
I'm having trouble trying to figure out how to adjust a timestamp array by +/- 12 hours.. pyarrow.compute.add() should work, but I can't figure out how it should handle timestamps.
import pyarrow.compute as pc
_ts = ["9/03/2023 00:35", "9/03/2023 12:35", "9/03/2023 6:35", "9/03/2023 18:35"]
_format = "%d/%m/%Y %H:%M"
timestamps = pc.strptime(_ts, format= _format, unit='s')
timestamps
<pyarrow.lib.TimestampArray object at 0x000001D04E4B6640>
[
2023-03-09 00:35:00,
2023-03-09 12:35:00,
2023-03-09 06:35:00,
2023-03-09 18:35:00
]
import pyarrow as pa
adjust_by_hours = pa.array([12,0,0,-12])
adjust_by_hours
<pyarrow.lib.Int64Array object at 0x000001D03FE78A00>
[
12,
0,
0,
-12
]
The final result I'm looking for is:
<pyarrow.lib.TimestampArray object at 0x000001D04E4B6640>
[
2023-03-09 12:35:00,
2023-03-09 12:35:00,
2023-03-09 06:35:00,
2023-03-09 06:35:00
]
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2023 BlackRock, Inc. All rights reserved.
Re: How to adjust pyarrow timestamps using pyarrow.compute
Posted by Felipe Oliveira Carvalho <fe...@gmail.com>.
You can add `duration` arrays to `timestamp` arrays to get new `timestamp`
arrays [1][2].
import pyarrow as pa
import pyarrow.compute as pc
_ts = ["9/03/2023 00:35", "9/03/2023 12:35", "9/03/2023 6:35", "9/03/2023
18:35"]
_format = "%d/%m/%Y %H:%M"
timestamps = pc.strptime(_ts, format= _format, unit='s')
deltas = pa.array([12 * 3660, 0, 0, -12 * 3600], type=pa.duration('s'))
pc.add(timestamps, deltas)
<pyarrow.lib.TimestampArray object at 0x7f5d44a2efe0>
[
2023-03-09 12:47:00,
2023-03-09 12:35:00,
2023-03-09 06:35:00,
2023-03-09 06:35:00
]
[1]
https://github.com/apache/arrow/blob/main/cpp/src/arrow/compute/kernels/scalar_arithmetic.cc#L1300
[2]
https://arrow.apache.org/docs/python/generated/pyarrow.duration.html#pyarrow.duration
--
Felipe
On Fri, Jul 21, 2023 at 1:29 PM Lee, David (PAG) <Da...@blackrock.com>
wrote:
> Resending.. somehow my code copying and pasting got mangled by ms outlook.
>
> -----Original Message-----
> From: Lee, David (PAG)
> Sent: Friday, July 21, 2023 9:25 AM
> To: 'user@arrow.apache.org' <us...@arrow.apache.org>
> Subject: How to adjust pyarrow timestamps using pyarrow.compute
>
> I'm having trouble trying to figure out how to adjust a timestamp array by
> +/- 12 hours.. pyarrow.compute.add() should work, but I can't figure out
> how it should handle timestamps.
>
> import pyarrow.compute as pc
>
> _ts = ["9/03/2023 00:35", "9/03/2023 12:35", "9/03/2023 6:35", "9/03/2023
> 18:35"]
> _format = "%d/%m/%Y %H:%M"
>
> timestamps = pc.strptime(_ts, format= _format, unit='s')
> timestamps
>
> <pyarrow.lib.TimestampArray object at 0x000001D04E4B6640> [
> 2023-03-09 00:35:00,
> 2023-03-09 12:35:00,
> 2023-03-09 06:35:00,
> 2023-03-09 18:35:00
> ]
>
> import pyarrow as pa
> adjust_by_hours = pa.array([12,0,0,-12])
> adjust_by_hours
>
> <pyarrow.lib.Int64Array object at 0x000001D03FE78A00> [
> 12,
> 0,
> 0,
> -12
> ]
>
> The final result I'm looking for is:
> <pyarrow.lib.TimestampArray object at 0x000001D04E4B6640> [
> 2023-03-09 12:35:00,
> 2023-03-09 12:35:00,
> 2023-03-09 06:35:00,
> 2023-03-09 06:35:00
> ]
>
>
>
>
> This message may contain information that is confidential or privileged.
> If you are not the intended recipient, please advise the sender immediately
> and delete this message. See
> http://www.blackrock.com/corporate/compliance/email-disclaimers for
> further information. Please refer to
> http://www.blackrock.com/corporate/compliance/privacy-policy for more
> information about BlackRock’s Privacy Policy.
>
>
> For a list of BlackRock's office addresses worldwide, see
> http://www.blackrock.com/corporate/about-us/contacts-locations.
>
> © 2023 BlackRock, Inc. All rights reserved.
>
RE: How to adjust pyarrow timestamps using pyarrow.compute
Posted by "Lee, David (PAG)" <Da...@blackrock.com>.
Resending.. somehow my code copying and pasting got mangled by ms outlook.
-----Original Message-----
From: Lee, David (PAG)
Sent: Friday, July 21, 2023 9:25 AM
To: 'user@arrow.apache.org' <us...@arrow.apache.org>
Subject: How to adjust pyarrow timestamps using pyarrow.compute
I'm having trouble trying to figure out how to adjust a timestamp array by +/- 12 hours.. pyarrow.compute.add() should work, but I can't figure out how it should handle timestamps.
import pyarrow.compute as pc
_ts = ["9/03/2023 00:35", "9/03/2023 12:35", "9/03/2023 6:35", "9/03/2023 18:35"]
_format = "%d/%m/%Y %H:%M"
timestamps = pc.strptime(_ts, format= _format, unit='s')
timestamps
<pyarrow.lib.TimestampArray object at 0x000001D04E4B6640> [
2023-03-09 00:35:00,
2023-03-09 12:35:00,
2023-03-09 06:35:00,
2023-03-09 18:35:00
]
import pyarrow as pa
adjust_by_hours = pa.array([12,0,0,-12])
adjust_by_hours
<pyarrow.lib.Int64Array object at 0x000001D03FE78A00> [
12,
0,
0,
-12
]
The final result I'm looking for is:
<pyarrow.lib.TimestampArray object at 0x000001D04E4B6640> [
2023-03-09 12:35:00,
2023-03-09 12:35:00,
2023-03-09 06:35:00,
2023-03-09 06:35:00
]
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2023 BlackRock, Inc. All rights reserved.