You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Rafferty97 (via GitHub)" <gi...@apache.org> on 2026/02/23 11:23:15 UTC
[I] Support CSV files encoded with charsets other than UTF-8 [arrow-rs]
Rafferty97 opened a new issue, #9465:
URL: https://github.com/apache/arrow-rs/issues/9465
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
It seems that CSV files encoded with non-UTF-8 charsets, such as Windows-1252, are annoyingly common in the wild. It would be useful to be able to consume them directly via an additional configuration option.
**Describe the solution you'd like**
Add a configuration option to the CSV reader to specify a character encoding, defaulting to UTF-8. The implementation can make us of `encoding_rs`, and could be feature-gated so as to not affect users who don't need this functionality.
**Describe alternatives you've considered**
The only alternative I can think of is to decode the entire CSV file up front before reading it via Apache Arrow, but this is suboptimal for a lot of usecases.
**Additional context**
I originally opened a similar issue in the Datafusion project, but after further reflection, figured it was possibly better implemented in Arrow itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files encoded with charsets other than UTF-8 [arrow-rs]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 closed issue #9465: Support CSV files encoded with charsets other than UTF-8
URL: https://github.com/apache/arrow-rs/issues/9465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files encoded with charsets other than UTF-8 [arrow-rs]
Posted by "brunal (via GitHub)" <gi...@apache.org>.
brunal commented on issue #9465:
URL: https://github.com/apache/arrow-rs/issues/9465#issuecomment-3976683283
I think this can be achieved without any arrow change, using `encoding_rs_*` streaming encoders/decoders.
````
// Basic
fn read_utf8(filename: &str, schema: SchemaRef) -> RecordBatch {
let file = File::open(filename).unwrap();
let mut csv = ReaderBuilder::new(Arc::new(schema)).build(file).unwrap();
let batch = csv.next().unwrap().unwrap();
batch
}
// With a different encoding
fn read_other_encoding(filename: &str, schema: SchemaRef, encoding: &encoding_rs::Encoding) -> RecordBatch {
let file = File::open(filename).unwrap();
let mut decoded = encoding_rs_rw::DecodingReader(file, encoding.new_decoder());
let mut csv = ReaderBuilder::new(Arc::new(schema)).build(decoded).unwrap();
let batch = csv.next().unwrap().unwrap();
batch
}
```
The change is minimal and doesn't require upfront decoding of the whole file. I believe that's a better solution than adding a crate feature (I don't like crate features) & another dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files encoded with charsets other than UTF-8 [arrow-rs]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 closed issue #9465: Support CSV files encoded with charsets other than UTF-8
URL: https://github.com/apache/arrow-rs/issues/9465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support CSV files encoded with charsets other than UTF-8 [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9465:
URL: https://github.com/apache/arrow-rs/issues/9465#issuecomment-4084934564
`label_issue.py` automatically added labels {'arrow'} from #9468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org