[jira] [Resolved] (ARROW-5089) [C++/Python] Writing dictionary encoded columns to parquet is extremely slow when using chunk size

["Wes McKinney (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney resolved ARROW-5089.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 0.15.0

This is resolved in master through my patches related to writing DictionaryArray to Parquet

0.14.1:

{code}
In [2]: timeit pq.write_table(table, pa.BufferOutputStream(), chunk_size=100)                               
3.18 s ± 16.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: timeit pq.write_table(table, pa.BufferOutputStream())                                               
6.28 ms ± 120 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
{code}

master (to be 0.15.0)

{code}
In [3]: timeit pq.write_table(table, pa.BufferOutputStream(), chunk_size=100)              
27.5 ms ± 2.49 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [4]: timeit pq.write_table(table, pa.BufferOutputStream())                              
5.8 ms ± 22.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
{code}

Indeed, a difference of about 100x

> [C++/Python] Writing dictionary encoded columns to parquet is extremely slow when using chunk size
> --------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-5089
>                 URL: https://issues.apache.org/jira/browse/ARROW-5089
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++
>    Affects Versions: 0.13.0
>            Reporter: Florian Jetter
>            Assignee: Wes McKinney
>            Priority: Major
>              Labels: parquet, performance
>             Fix For: 0.15.0
>
>
> Currently, there is a workaround for dict encoded columns in place to handle writing dict encoded columns to parquet.
> The workaround converts the dict encoded array to its plain version before writing to parquet. This is painfully slow since for every row group the entire array is converted over and over again.
> The following example is orders of magnitude slower than the non-dict encoded version:
> {code}
> import pyarrow as pa
> import pyarrow.parquet as pq
> import pandas as pd
> df = pd.DataFrame({"col": ["A", "B"] * 100000}).astype("category")
> table = pa.Table.from_pandas(df)
> buf = pa.BufferOutputStream()
> pq.write_table(
>     table,
>     buf,
>     chunk_size=100,
> )
>  {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)