You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/04 14:11:24 UTC

[GitHub] [arrow-datafusion] mvanschellebeeck opened a new issue, #4502: Support writing lists in the arrow csv writer

mvanschellebeeck opened a new issue, #4502:
URL: https://github.com/apache/arrow-datafusion/issues/4502

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   Add support to the arrow CSV writer for writing lists.
   
   Currently an sqllogictest of the format:
   ```
   query T
   SELECT array_agg(c13) FROM (SELECT * FROM aggregate_test_100 ORDER BY c13 LIMIT 2) test
   ----
   [0VVIHzxWtNOFLtnhjHEKjXaJOSLJfm0keZ5G8BffGwgF2RwQD59TFzMStxCB]
   ```
   
   will fail with 
   `thread 'main' panicked at 'called Result::unwrap() on an Err value: CsvError("CSV Writer does not support List(Field { name: \"item\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) data type")', datafusion/core/tests/sqllogictests/src/main.rs:134:33`
   
   Source of failure in arrow-csv: https://github.com/apache/arrow-rs/blob/master/arrow-csv/src/writer.rs#L228
   
   Not sure if this makes sense to implement upstream (in arrow-csv) or as part of datafusions test harness
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] tustvold closed issue #4502: Support writing lists in the arrow csv writer

Posted by "tustvold (via GitHub)" <gi...@apache.org>.

tustvold closed issue #4502: Support writing lists in the arrow csv writer
URL: https://github.com/apache/arrow-datafusion/issues/4502


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] melgenek commented on issue #4502: Support writing lists in the arrow csv writer

Posted by "melgenek (via GitHub)" <gi...@apache.org>.

melgenek commented on issue #4502:
URL: https://github.com/apache/arrow-datafusion/issues/4502#issuecomment-1401653984

   Sqllogictest doesn't use the CSV writer anymore https://github.com/apache/arrow-datafusion/pull/4578.
   The test from this issue passes successfully https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/test_files/aggregate.slt#L953-L957.
   
   @xudong963 @alamb @tustvold I guess, this ticket can be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] tustvold commented on issue #4502: Support writing lists in the arrow csv writer

Posted by GitBox <gi...@apache.org>.

tustvold commented on issue #4502:
URL: https://github.com/apache/arrow-datafusion/issues/4502#issuecomment-1341375199

   This would be an upstream issue in arrow-rs if we wish to add support for this, and I would be happy to review a PR adding support for it.
   
   That being said, I'm not sure how to encode nested data in CSV? I had understood the format to be strictly tabular? The docs for [arrow python](https://arrow.apache.org/docs/python/csv.html#reading-and-writing-csv-files) would appear to suggest only primitive types are supported although I haven't tested this?
   
   FWIW arrow-rs supports reading/writing nested JSON


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org