[jira] [Commented] (ARROW-7655) [Python] csv.ConvertOptions Do Not Pass Through/Retain Nullability from Schema

["Tim Lantz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021293#comment-17021293 ] 

Tim Lantz commented on ARROW-7655:
----------------------------------

Re: my side note above, I filed https://issues.apache.org/jira/browse/ARROW-7656 as well. I see that in ARROW-6536 there is discussion on why in the C++ API you need to set both and that makes perfect sense so this is just a documentation thing.

> [Python] csv.ConvertOptions Do Not Pass Through/Retain Nullability from Schema
> ------------------------------------------------------------------------------
>
>                 Key: ARROW-7655
>                 URL: https://issues.apache.org/jira/browse/ARROW-7655
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.15.1
>         Environment: Reproduced on Ubuntu 18.04 and OSX Catalina in Python 3.7.4.
>            Reporter: Tim Lantz
>            Priority: Major
>              Labels: CSV
>
>  
> Originally mentioned in: [https://github.com/apache/arrow/issues/6243]
> *High level description of the issue:*
>  * It is possible ([though not documented|https://issues.apache.org/jira/browse/ARROW-7654]) that you may assign the column_types field of ConvertOptions to a Schema object instead of a Dict[str, DataType].
>  * Expected result: the nullable attribute, in addition to the type, of the Fields in the Schema supplied are present on the Schema used when reading CSV data.
>  * Actual result: the Field type information is present, but nullable is lost. All fields are nullable.
> *Minimal reproduction case:*
>  * Use case notes: this is especially noticeable when using pyarrow as a meant to save data with a known schema to parquet as the ParquetWriter will check that the schema of a table being written matches the schema supplied to the writer. If that same schema is used to to read the CSV data and contains a nullable field, a mismatch will be detected resulting in an error which is demonstrated below.
>  
> {code:java}
> $ cat test.csv 
> 0
> 1
> $ python
> >>> import pyarrow
> >>> schema = pyarrow.schema([pyarrow.field(name="foo", type=pyarrow.bool_(), nullable=False)])
> >>> read_options = csv.ReadOptions(column_names=["foo"])
> >>> from pyarrow import csv
> >>> read_options = csv.ReadOptions(column_names=["foo"])
> >>> convert_options = csv.ConvertOptions(column_types=schema)
> >>> table = csv.read_csv("test.csv", convert_options=convert_options, read_options=read_options)
> >>> schema
> foo: bool not null
> >>> table.schema
> foo: bool
> >>> from pyarrow import parquet as pq
> >>> writer = pq.ParquetWriter("test.parquet", schema)
> >>> writer.write_table(table)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "(REDACTED)/lib/python3.7/site-packages/pyarrow-0.15.1-py3.7-macosx-10.9-x86_64.egg/pyarrow/parquet.py", line 472, in write_table
>     raise ValueError(msg)
> ValueError: Table schema does not match schema used to create file: 
> table:
> foo: bool vs. 
> file:
> foo: bool not null
> >>> pyarrow.__version__
> '0.15.1'
> >>> exit()
> $ python --version
> Python 3.7.4{code}
>  
>  * As a side note: if I don't set column_names in read_options when calling read_csv, but I set convert_options with column_types set, type inference is still performed which seems like a bug vs. what the docs state. That seems like a possibly related, but independent bug, and I haven't done a search yet to see if it is an open/known issue but if someone believes it should be filed with a repro case upon reading this I am happy to help! I only realized this when minimizing the repro case as my original code was setting column_names.
> *Potential source of issue:*
>  * **I did not yet look at how hard it is to fix, but I note that [here|https://github.com/apache/arrow/blob/ace72c2afa6b7608bca9ba858fdd10b23e7f2dbf/python/pyarrow/_csv.pyx#L411] only the name and type are passed down from a Field.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)