You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2020/09/10 13:45:00 UTC
[jira] [Assigned] (ARROW-9962) [Python] Conversion to pandas with
index column using fixed timezone fails
[ https://issues.apache.org/jira/browse/ARROW-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joris Van den Bossche reassigned ARROW-9962:
--------------------------------------------
Assignee: Joris Van den Bossche
> [Python] Conversion to pandas with index column using fixed timezone fails
> --------------------------------------------------------------------------
>
> Key: ARROW-9962
> URL: https://issues.apache.org/jira/browse/ARROW-9962
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Joris Van den Bossche
> Assignee: Joris Van den Bossche
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From https://github.com/pandas-dev/pandas/issues/35997: it seems we are handling a normal column and index column differently in the conversion to pandas.
> {code}
> In [5]: import pandas as pd
> ...: from datetime import datetime, timezone
> ...:
> ...: df = pd.DataFrame([[datetime.now(timezone.utc), datetime.now(timezone.utc)]], columns=['date_index', 'date_column'])
> ...: table = pa.Table.from_pandas(df.set_index('date_index'))
> ...:
> In [6]: table
> Out[6]:
> pyarrow.Table
> date_column: timestamp[ns, tz=+00:00]
> date_index: timestamp[ns, tz=+00:00]
> In [7]: table.to_pandas()
> ...
> UnknownTimeZoneError: '+00:00'
> {code}
> So this happens specifically for "fixed offset" timezones, and only for index columns (eg {{table.select(["date_column"]).to_pandas()}} works fine).
> It seems this is because for columns we use our helper {{make_tz_aware}} to convert the string "+01:00" to a python timezone, which is then understood by pandas (the string is not handled by pandas). But for the index column we fail to do this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)