[jira] [Commented] (ARROW-1328) [Python] pyarrow.Table.from_pandas option timestamps_to_ms changes column values

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113105#comment-16113105 ] 

Wes McKinney commented on ARROW-1328:
-------------------------------------

Marked for 0.6.0. I would like to do away with the {{timestamps_to_ms}} argument altogether and add a writer option in parquet-cpp to cast nanoseconds (or microseconds) to milliseconds

> [Python] pyarrow.Table.from_pandas option timestamps_to_ms changes column values
> --------------------------------------------------------------------------------
>
>                 Key: ARROW-1328
>                 URL: https://issues.apache.org/jira/browse/ARROW-1328
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.5.0
>         Environment: MacOS 10.12.6
> python 2.7.12
> pyarrow==0.5.0
> pandas==0.20.3
> numpy==1.13.1
> python-dateutil==2.6.1
>            Reporter: Florian Jetter
>             Fix For: 0.6.0
>
>
> The {{pyarrow.Table.from_pandas}} option {{timestamps_to_ms}} changes the value of the datetime column
> {code:none}
> >>> import pyarrow as pa
> >>> import pandas as pd
> >>> from datetime import datetime
> >>>
> >>> df = pd.DataFrame({'datetime': [datetime(2017, 1, 1)]})
> >>> dtype = df['datetime'].dtype
> >>> pa_type = pa.from_numpy_dtype(dtype)
> >>> schema = pa.schema([pa.field('datetime', pa_type)])
> >>> table = pa.Table.from_pandas(df=df, schema=schema, timestamps_to_ms=True)
> >>> table.to_pandas()
>                     datetime
> 0 1970-01-01 00:24:43.228800
> >>> dtype
> dtype('<M8[ns]')
> >>> pa_type
> TimestampType(timestamp[ns])
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)