You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "rgoubet (via GitHub)" <gi...@apache.org> on 2023/07/04 15:00:26 UTC

[GitHub] [arrow] rgoubet opened a new issue, #36459: Specify encoding for pyarrow.csv.WriteOptions

rgoubet opened a new issue, #36459:
URL: https://github.com/apache/arrow/issues/36459

   ### Describe the enhancement requested
   
   Apparently, there is no `encoding` option in the `pyarrow.csv.WriteOptions` class, so there seems to be no way to specify an encoding when writing a CSV file.
   
   There might be cases where `UTF-8` is not wanted. The most obvious one to me is to write using `UTF-8-SIG` for full compatibility with Excel.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Python] Add an option to specify encoding in WriteOptions (CSV Writer) [arrow]

Posted by "thisisnic (via GitHub)" <gi...@apache.org>.

thisisnic commented on issue #36459:
URL: https://github.com/apache/arrow/issues/36459#issuecomment-3921766568

   This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] AlenkaF commented on issue #36459: Specify encoding for pyarrow.csv.WriteOptions

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.

AlenkaF commented on issue #36459:
URL: https://github.com/apache/arrow/issues/36459#issuecomment-1621281683

   Yes, there is no `encoding` option for the `WriteOptions` of CSV Writer in the C++ therefore it is not present in pyarrow
   https://arrow.apache.org/docs/dev/cpp/api/formats.html#_CPPv4N5arrow3csv12WriteOptionsE.
   
   Will change this to an enhancement request for Arrow C++ CSV Writer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] westonpace commented on issue #36459: [C++][Python] Add an option to specify encoding in WriteOptions (CSV Writer)

Posted by "westonpace (via GitHub)" <gi...@apache.org>.

westonpace commented on issue #36459:
URL: https://github.com/apache/arrow/issues/36459#issuecomment-1631602685

   Thanks Alenka.  I agree this will need C++ changes.  Note that handling different encodings on the read path is not actually a part of the CSV code.  Instead we have a `TransformInputStream` which is an input stream with some arbitrary transformation applied to it.  Then, in pyarrow, we have an implementation of `TransformInputStream` which applies the decode-on-read.
   
   This did end up being a little inconvenient for datasets since the user doesn't create the input stream.  So we ended up allowing file formats to specify a stream transform function that we would apply on every file we read.
   
   So if we want an equivalent feature on the write path we would need:
   
    * Create a TransformOutputStream version of output stream
    * Create pyarrow implementation of TransformOutputStream that applies the encoding
    * Add a stream_transform_func to the dataset writer
   
   On the other hand we could just make encoding a property of the CSV writer as Alenka suggested.  This would probably be simpler in the long run and, at the moment, CSV is the only file format that has an flexibility on encoding.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org