[jira] [Commented] (ARROW-5419) [C++] CSV strings_can_be_null option doesn't respect all null_values

["Joris Van den Bossche (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-5419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848741#comment-16848741 ] 

Joris Van den Bossche commented on ARROW-5419:
----------------------------------------------

As a sidenote: an "empty" field is the default in pandas' {{to_csv}} to represent missing values. So it would indeed be good to support this when reading back in. 

Related to this I was thinking whether we want to differentiate between empty field and actual quoted empty string, but I suppose this is not that easy (as once parsed, both are identical?). Also pandas' read_csv does not seem to differentiate.

> [C++] CSV strings_can_be_null option doesn't respect all null_values
> --------------------------------------------------------------------
>
>                 Key: ARROW-5419
>                 URL: https://issues.apache.org/jira/browse/ARROW-5419
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, Python
>         Environment: Python 3.6.8
> PyArrow 0.13.1.dev225+g184b8deb
> NumPy 1.16.3
> Pandas 0.24.2
>            Reporter: Dennis Waldron
>            Priority: Minor
>              Labels: csv
>
> Relates to ARROW-5195 and [https://github.com/apache/arrow/issues/4184]
> I was testing the new *strings_can_be_null* ConvertOption (built from git 184b8deb651c6f6308c0fa2a595f5a40f5da8ce8) in conjunction with the CSV reader and noted that when enabled and an empty string is parsed that it doesn't return NULL despite '' being in the default null_values list ([https://github.com/apache/arrow/blob/f7ef65e5fc367f1f5649dfcea0754e413fcca394/cpp/src/arrow/csv/options.cc#L28)]
> {code:java}
> options.null_values = {"", "#N/A", "#N/A N/A", "#NA", "-1.#IND", "-1.#QNAN",
> "-NaN", "-nan", "1.#IND", "1.#QNAN", "N/A", "NA",
> "NULL", "NaN", "n/a", "nan", "null"};
> {code}
> Given that the *strings_can_be_null* option was added to expose the same NULL processing functionality with respect to strings as *pandas.read_csv,* I believe that it should also be able to handle empty strings. ** 
> In Pandas:
> {code:java}
> content = b"a,b\n1,null\n2,\n3,test"
> df = pd.read_csv(io.BytesIO(content))
> print(df)
>    a     b
> 0  1   NaN
> 1  2   NaN
> 2  3  test
> {code}
> In PyArrow:
> {code:java}
> convert_options = pc.ConvertOptions(strings_can_be_null=True)
> table = pc.read_csv(io.BytesIO(content), convert_options=convert_options)
> print(table.to_pydict())
> OrderedDict([('a', [1, 2, 3]), ('b', [None, '', 'test'])])
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)