You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Maxim Gekk (JIRA)" <ji...@apache.org> on 2018/02/14 22:21:00 UTC
[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in
charset different from UTF-8
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364849#comment-16364849 ]
Maxim Gekk edited comment on SPARK-23410 at 2/14/18 10:20 PM:
--------------------------------------------------------------
[~bersprockets] does your json contain BOM in the first 2 bytes? By using the BOM, jackson detects encoding: https://github.com/FasterXML/jackson-core/blob/2.6/src/main/java/com/fasterxml/jackson/core/json/ByteSourceJsonBootstrapper.java#L110-L173
was (Author: maxgekk):
[~bersprockets] does your json contain BOM in the first 2 bytes?
> Unable to read jsons in charset different from UTF-8
> ----------------------------------------------------
>
> Key: SPARK-23410
> URL: https://issues.apache.org/jira/browse/SPARK-23410
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
> Affects Versions: 2.3.0
> Reporter: Maxim Gekk
> Priority: Major
>
> Currently the Json Parser is forced to read json files in UTF-8. Such behavior breaks backward compatibility with Spark 2.2.1 and previous versions that can read json files in UTF-16, UTF-32 and other encodings due to using of the auto detection mechanism of the jackson library. Need to give back to users possibility to read json files in specified charset and/or detect charset automatically as it was before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org