[jira] [Created] (ARROW-16682) [Python] CSV reader: allow parsing without encoding errors

["Thomas Buhrmann (Jira)" <ji...@apache.org>]

Thomas Buhrmann created ARROW-16682:
---------------------------------------

             Summary: [Python] CSV reader: allow parsing without encoding errors
                 Key: ARROW-16682
                 URL: https://issues.apache.org/jira/browse/ARROW-16682
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Python
    Affects Versions: 8.0.0
            Reporter: Thomas Buhrmann


When trying to read arbitrary CSV files, it is not possible to infer/guess the correct encoding 100% of the time. The Arrow CSV reader will currently fail if any byte cannot be decoded given the specified encoding (see example below).

With pandas.read_csv(), I can often get a result that is 99.9% correct by passing it a text stream decoded in Python with [errors="replace"|https://docs.python.org/3/library/codecs.html#error-handlers] (or "ignore" etc.).

Pyarrow's csv.read_csv() on the other hand neither accepts an already decoded text stream (TypeError: binary file expected, got text file), nor a parameter to configure what to do with decoding errors. As a result the parser simply fails.

The simplest solution would probably be to expose Python's error handling in pyarrow.csv.ReadOptions (e.g. encoding_errors: "strict" | "ignore" | "replace" ...).

It would also be useful to document the behaviour of the CSV reader. E.g. that it only accepts binary streams, and how encoding errors are handled. In particular it is unclear what "Columns that cannot decode using this encoding can still be read as Binary" means, since the parser will currently fail if any bytes cannot be decoded.

Toy example:

 
{code:java}
import io

txt = """
col_😀_1, col2
0,a
1,b
"""
buffer = io.BytesIO(txt.encode("utf-8"))
pa.csv.read_csv(buffer, pa.csv.ReadOptions(encoding="ascii")){code}
{noformat}
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 5: ordinal not in range(128){noformat}
whereas "with pandas":
{code:java}
buffer = io.BytesIO(txt.encode("utf-8"))
text = io.TextIOWrapper(buffer, encoding="ascii", errors="replace")
pd.read_csv(text){code}
{noformat}
   col_����_1  col2
0           0     a
1           1     b
{noformat}
 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)