You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Jone Zhang <jo...@gmail.com> on 2017/04/06 04:47:07 UTC

Why chinese character gash appear when i use spark textFile?

var textFile = sc.textFile("xxx");
textFile.first();
res1: String = 1.0     100733314       18_?????:100733314
8919173c6d49abfab02853458247e584        1:129:18_?????:1.0


hadoop fs -cat xxx
1.0    100733314       18_百度输入法:100733314 8919173c6d49abfab02853458247e584
     1:129:18_百度输入法:1.0

Why  chinese character gash appear when i use spark textFile?
The code of hdfs file is utf-8.


Thanks

Re: Why chinese character gash appear when i use spark textFile?

Posted by "颜发才 (Yan Facai)" <fa...@gmail.com>.

Perhaps your file is not utf-8.

I cannot reconstruct it.

### HADOOP:
~/Downloads ❯❯❯ hdfs -cat hdfs://xxxx/test.txt
17/04/06 13:43:58 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
1.0     862910025238798 100733314       18_百度输入法:100733314
8919173c6d49abfab02853458247e584        1:129:18_百度输入法:1.0%

### SPARK
scala> val t = sc.textFile("hdfs://xxxx/test.txt")
t: org.apache.spark.rdd.RDD[String] = hdfs://xxxx/test.txt
MapPartitionsRDD[3] at textFile at <console>:24

scala> t.first
res2: String = 1.0     862910025238798 100733314       18_百度输入法:100733314






On Thu, Apr 6, 2017 at 12:47 PM, Jone Zhang <jo...@gmail.com> wrote:

> var textFile = sc.textFile("xxx");
> textFile.first();
> res1: String = 1.0     100733314       18_?????:100733314
> 8919173c6d49abfab02853458247e584        1:129:18_?????:1.0
>
>
> hadoop fs -cat xxx
> 1.0    100733314       18_百度输入法:100733314 8919173c6d49abfab02853458247e584
>      1:129:18_百度输入法:1.0
>
> Why  chinese character gash appear when i use spark textFile?
> The code of hdfs file is utf-8.
>
>
> Thanks
>