You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "davlee1972 (via GitHub)" <gi...@apache.org> on 2024/03/19 05:37:26 UTC
[I] Add compression option to pyarrow.csv.WriteOptions [arrow]
davlee1972 opened a new issue, #40658:
URL: https://github.com/apache/arrow/issues/40658
### Describe the enhancement requested
Can we support gzip compression with csv writing? This would also apply to pyarrow.dataset.write_dataset.
The default extension when writing text files should also reflect the compression format.. I.e. abc.csv.gz and abc.csv.zip
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Add compression option to pyarrow.csv.WriteOptions [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #40658:
URL: https://github.com/apache/arrow/issues/40658#issuecomment-3547492866
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this usage question has evolved into a feature request or docs update, please remove the 'Type: usage' label and add the 'Type: enhancement' label instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Add compression option to pyarrow.csv.WriteOptions [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #40658: [Python] Add compression option to pyarrow.csv.WriteOptions
URL: https://github.com/apache/arrow/issues/40658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Add compression option to pyarrow.csv.WriteOptions [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #40658:
URL: https://github.com/apache/arrow/issues/40658#issuecomment-2009120646
It's not built in into the csv writer itself (for the reading side it is), but you can let it compress on the fly using `CompressedOutputStream`. See the usage example at https://arrow.apache.org/docs/python/csv.html#usage:
```python
import pyarrow as pa
import pyarrow.csv as csv
with pa.CompressedOutputStream("tips.csv.gz", "gzip") as out:
csv.write_csv(table, out)
```
That's only the direct CSV writer, and doesn't work through `pyarrow.dataset`. For compressed CSV in writing datasets we have an open feature request: https://github.com/apache/arrow/issues/34085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org