You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "connec (via GitHub)" <gi...@apache.org> on 2024/07/16 13:54:24 UTC
[I] Add support for `newlines_in_values` [arrow-rs]
connec opened a new issue, #6066:
URL: https://github.com/apache/arrow-rs/issues/6066
### Is your feature request related to a problem or challenge?
I'm trying to read CSVs that include newlines in (quoted) values.
### Describe the solution you'd like
Some googling revealed that this isn't supported currently by the `arrow-csv` crate, whereas that functionality does exist in the C++ ([`ParseOptions::newlines_in_values`](https://arrow.apache.org/docs/cpp/api/formats.html#_CPPv4N5arrow3csv12ParseOptions18newlines_in_valuesE)) and Python ([`ParseOptions.newlines_in_values`](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ParseOptions.html#pyarrow.csv.ParseOptions.newlines_in_values)) implementations.
Ideally, a `newlines_in_values` field could be added to [`arrow_csv::reader::Format`](https://docs.rs/arrow-csv/latest/arrow_csv/reader/struct.Format.html) to support this functionality.
Note that the Python docs call out the performance implications of this:
> Setting this to True reduces the performance of multi-threaded CSV reading.
I haven't dug into the implementation, but I don't think `arrow-rs` supports parallel CSV reading itself? So this implication might not hold at this layer (my original use-case is with `datafusion` where this would be relevant).
### Describe alternatives you've considered
The only alternative I can see would be to preprocess the CSV before feeding it to arrow. I haven't explored this option as my use-case is through datafusion and this would require a lot of plumbing, and it seems valuable to have parity with other arrow CSV packages (C++ and Python, at least).
### Additional context
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for `newlines_in_values` [arrow-rs]
Posted by "connec (via GitHub)" <gi...@apache.org>.
connec closed issue #6066: Add support for `newlines_in_values`
URL: https://github.com/apache/arrow-rs/issues/6066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for `newlines_in_values` [arrow-rs]
Posted by "connec (via GitHub)" <gi...@apache.org>.
connec commented on issue #6066:
URL: https://github.com/apache/arrow-rs/issues/6066#issuecomment-2234596575
You're exactly right, my bad – I made an assumption from the lack of an option controlling it as a lack of support.
My use case is with datafusion, and I can probably achieve what I want by limiting the parallelism of their reader down to one.
Apologies for the noise!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for `newlines_in_values` [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6066:
URL: https://github.com/apache/arrow-rs/issues/6066#issuecomment-2233724294
I believe this should just work, is there a particular issue you are running into?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org