You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2019/03/02 20:51:00 UTC
[jira] [Commented] (SPARK-26016) Encoding not working when using a
map / mapPartitions call
[ https://issues.apache.org/jira/browse/SPARK-26016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782520#comment-16782520 ]
Sean Owen commented on SPARK-26016:
-----------------------------------
Yeah, the TL;DR from reading the code is that it writes UTF-8, right?
df.write.text always writes UTF-8 strings; I think that's the most immediate explanation for all of this. df.read.text also only reads UTF-8, as it pushes down to the Hadoop input formats and Text works in terms of UTF-8. I think it's working insofar as ISO-8859-1's character mapping is a subset of UTF-8's. I'm not as sure why the write path doesn't accidentally work; BOMs? I think this is 80% right at least as an explanation.
The problem is that this isn't documented anywhere I can find. I think people are just accustomed to always working in UTF-8 and the fact that you can read some common encodings as UTF-8 anyway. That much can easily be documented.
It's exacerbated by the fact that there's an 'encoding' option on DataFrameReader/Writer, which does nothing but affect how the line separator is treated. (I don't quite get why you could use a non-UTF-8 encoding of your line separator if everything else assumes UTF-8!)
[~maxgekk] it looks like this option was added in https://github.com/apache/spark/pull/20937/files#diff-cbd2ad0c69c1517fbbc42b587fd37e25R44 ; was it necessary for the text data source? for the reason above I'm not sure about how it works. I see it's also used for JSON, separately. Is it meant to be 'hidden'?
> Encoding not working when using a map / mapPartitions call
> ----------------------------------------------------------
>
> Key: SPARK-26016
> URL: https://issues.apache.org/jira/browse/SPARK-26016
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.4.0
> Reporter: Chris Caspanello
> Priority: Major
> Attachments: spark-sandbox.zip
>
>
> Attached you will find a project with unit tests showing the issue at hand.
> If I read in a ISO-8859-1 encoded file and simply write out what was read; the contents in the part file matches what was read. Which is great.
> However, the second I use a map / mapPartitions function it looks like the encoding is not correct. In addition a simple collectAsList and writing that list of strings to a file does not work either. I don't think I'm doing anything wrong. Can someone please investigate? I think this is a bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org