You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Rafferty97 (via GitHub)" <gi...@apache.org> on 2026/02/22 01:25:54 UTC
[I] Support non-UTF-8 encoded CSV files [datafusion]
Rafferty97 opened a new issue, #20473:
URL: https://github.com/apache/datafusion/issues/20473
### Is your feature request related to a problem or challenge?
Currently, Datafusion doesn't appear to support reading CSV files that use a non-UTF-8 encoding scheme, such as the common ISO-8859-1 or others.
While CSV may be a terrible data format, it's also ubiquitous in the wild and many of them use alternative character encodings. It would be useful if there was an option to read CSV files that use an encoding other than UTF-8.
### Describe the solution you'd like
Add an option to `CsvOptions` or elsewhere to specify the encoding used by the input file, defaulting to `UTF-8`. Datafusion could then use `encoding_rs` internally to decode chunks of incoming data.
### Describe alternatives you've considered
An alternative to depending on `encoding_rs` directly would be to expose an option that allowed users to provide their own decoding logic, which they would then likely delegate to `encoding_rs`. This might be desirable if the added dependency is deemed to heavy (though it could easily be put behind a feature flag).
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support non-UTF-8 encoded CSV files [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #20473:
URL: https://github.com/apache/datafusion/issues/20473#issuecomment-3979011996
I have closed my PR in `arrow-csv` based on feedback from reviewers, as they would prefer that consumes of library perform their own decoding rather than extending its API. I will open a PR for this one soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support non-UTF-8 encoded CSV files [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #20473:
URL: https://github.com/apache/datafusion/issues/20473#issuecomment-3940380461
I have made a start on this work here: https://github.com/Rafferty97/datafusion/tree/non-utf8-csv
I'm not sure about my approach to feature flagging, or how to best add unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support non-UTF-8 encoded CSV files [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #20473:
URL: https://github.com/apache/datafusion/issues/20473#issuecomment-3952395911
I have opened a PR to incorporate this functionality into `arrow-csv`: https://github.com/apache/arrow-rs/pull/9468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support non-UTF-8 encoded CSV files [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #20473:
URL: https://github.com/apache/datafusion/issues/20473#issuecomment-3942480592
On second thought, I suspect this might be better implemented in `arrow-csv` itself, so I'm going to experiment with that option now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support non-UTF-8 encoded CSV files [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #20473:
URL: https://github.com/apache/datafusion/issues/20473#issuecomment-3979438656
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org