[jira] [Updated] (ARROW-25) [C++] Implement delimited file scanner / CSV reader

["Wes McKinney (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney updated ARROW-25:
------------------------------
    Summary: [C++] Implement delimited file scanner / CSV reader  (was: C++: Implement delimited file scanner / CSV reader)

> [C++] Implement delimited file scanner / CSV reader
> ---------------------------------------------------
>
>                 Key: ARROW-25
>                 URL: https://issues.apache.org/jira/browse/ARROW-25
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: C++
>            Reporter: Wes McKinney
>            Priority: Major
>
> Like Parquet and binary file formats, text files will be an important data medium for converting to and from in-memory Arrow data. 
> pandas has some (Apache-compatible) business logic we can learn from here (as one of the gold-standard CSV readers in production use)
> https://github.com/pydata/pandas/blob/master/pandas/src/parser/tokenizer.h
> https://github.com/pydata/pandas/blob/master/pandas/parser.pyx
> While very fast, this this should be largely written from scratch to target the Arrow memory layout, but we can reuse certain aspects like the tokenizer DFA (which originally came from the Python interpreter csv module implementation)
> https://github.com/pydata/pandas/blob/master/pandas/src/parser/tokenizer.c#L713



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)