[jira] [Resolved] (ARROW-4629) [Python] Pandas to arrow conversion slowed down by local imports

["Uwe L. Korn (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe L. Korn resolved ARROW-4629.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 0.12.1

Issue resolved by pull request 3706
[https://github.com/apache/arrow/pull/3706]

> [Python] Pandas to arrow conversion slowed down by local imports
> ----------------------------------------------------------------
>
>                 Key: ARROW-4629
>                 URL: https://issues.apache.org/jira/browse/ARROW-4629
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Florian Jetter
>            Assignee: Florian Jetter
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 0.13.0, 0.12.1
>
>         Attachments: image-2019-02-19-19-10-46-330.png
>
>          Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The pandas to arrow conversion is currently slowed down significantly by various local import statements.
> {code}
> import pandas as pd
> import pyarrow as pa
> import cProfile
> ser = pd.Series(range(10000))
> df = pd.DataFrame({col: ser.copy(deep=True) for col in range(50)})
> # Simulate a real dataset, i.e. force copy of data
> df = df.astype({col: str for col in range(25)})
> prof = cProfile.Profile()
> prof.enable()
> # a few times to collect statistics
> for _ in range(100):
>     pa.Table.from_pandas(df, nthreads=1)
> prof.disable()
> prof.dump_stats("array_conversion.prof")
> {code}
> !image-2019-02-19-19-10-46-330.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)