[jira] [Comment Edited] (ARROW-3722) [C++] Allow specifying column types to CSV reader

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680391#comment-16680391 ] 

Wes McKinney edited comment on ARROW-3722 at 11/8/18 9:14 PM:
--------------------------------------------------------------

You should have a look at various CSV libraries what API they expose for this. One way would be to have {{unordered_map<string, TypePtr>}}, so in Python you would indicate say {{{'col1': 'int32', 'col2': 'string'}}} and then when the column converters are initialized, they would use a type-restricted one rather than doing inference.

Some libraries allow you to indicate the column type by integer position in addition to the column name. 

Passing a whole schema is likely too labor intensive, especially for files with a lot of column names. Often, you have only a few problematic columns that you want to force a type while letting type inference run its course on the other columns

We'll eventually want to add the option to record statistics about unconvertible values (by computing the number of observations for each unconvertible value), and also to set them to null. I've found that people waste a lot of time debugging what's wrong with a bad CSV file when a column has some bad values. 


was (Author: wesmckinn):
You should have a look at various CSV libraries what API they expose for this. One way would be to have {{unordered_map<string, TypePtr>}}, so in Python you would indicate say {{{'col1': 'int32', 'col2': 'string'}}} and then when the column converters are initialized, they would use a type-restricted one rather than doing inference.

Some libraries allow you to indicate the column type by integer position in addition to the column name. 

Passing a whole schema is likely too labor intensive, especially for files with a lot of column names. Often, you have only a few problematic columns that you want to force a type while letting type inference run its course on the other columns

We'll eventually want to add the option to record statistics about unconvertible values (by computing the number of observations for each unconvertible value), and also to set them to null. I've found that people waste a lot of time debugging what's wrong with a bad CSV file when a column has some bad values. 

> [C++] Allow specifying column types to CSV reader
> -------------------------------------------------
>
>                 Key: ARROW-3722
>                 URL: https://issues.apache.org/jira/browse/ARROW-3722
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>    Affects Versions: 0.11.1
>            Reporter: Antoine Pitrou
>            Priority: Major
>
> I'm not sure how to expose this. The easiest, implementation-wise, would be to allow passing a {{Schema}} (for example inside the {{ConvertOptions}}).
> Another possibility is to allow specifying the default types for type inference. For example type inference currently infers integers as {{int64}}, but the user might prefer {{int32}}.
> Thoughts?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)