[jira] [Resolved] (ARROW-3408) [C++] Add option to CSV reader to dictionary encode individual columns or all string / binary columns

["Wes McKinney (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney resolved ARROW-3408.
---------------------------------
    Resolution: Fixed

Issue resolved by pull request 5785
[https://github.com/apache/arrow/pull/5785]

> [C++] Add option to CSV reader to dictionary encode individual columns or all string / binary columns
> -----------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-3408
>                 URL: https://issues.apache.org/jira/browse/ARROW-3408
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: C++, C++ - Dataset
>            Reporter: Wes McKinney
>            Assignee: Antoine Pitrou
>            Priority: Major
>              Labels: csv, dataset, pull-request-available
>             Fix For: 1.0.0
>
>          Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> For many datasets, dictionary encoding everything can result in drastically lower memory usage and subsequently better performance in doing analytics
> One difficulty of dictionary encoding in multithreaded conversions is that ideally you end up with one dictionary at the end. So you have two options:
> * Implement a concurrent hashing scheme -- for low cardinality dictionaries, the overhead associated with mutex contention will not be meaningful, for high cardinality it can be more of a problem
> * Hash each chunk separately, then normalize at the end
> My guess is that a crude concurrent hash table with a mutex to protect mutations and resizes is going to outperform the latter



--
This message was sent by Atlassian Jira
(v8.3.4#803005)