You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "yunzhi.lyz (JIRA)" <ji...@apache.org> on 2015/03/13 08:29:38 UTC
[jira] [Updated] (SPARK-6316) add a parameter for
SparkContext(conf).textFile() method , support for multi-language hdfs
file , e.g. "gbk"
[ https://issues.apache.org/jira/browse/SPARK-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
yunzhi.lyz updated SPARK-6316:
------------------------------
Description:
add a parameter for SparkContext(conf).textFile() method , support for multi-language hdfs file .
e.g. val file = new SparkContext(conf).textFile(args(0), 10,"gbk")
modify the code：
org.apache.spark.SparkContext
+ def defaultEncoding: String = "utf-8"
-- def textFile(path: String, minPartitions: Int = defaultMinPartitions): RDD[String] = {
hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
minPartitions).map(pair => pair._2.toString).setName(path)
}
++ def textFile(path: String, minPartitions: Int = defaultMinPartitions,encoding: String = defaultEncoding): RDD[String] = {
hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
minPartitions).map(pair => new String(pair._2.getBytes(), 0 , pair._2.getLength(), encoding)).setName(path)
}
was:
add a parameter for SparkContext(conf).textFile() method , support for multi-language hdfs file .
e.g. val file = new SparkContext(conf).textFile(args(0), 10,"gbk")
modify the code：
org.apache.spark.SparkContext
+ def defaultEncoding: String = "utf-8"
- def textFile(path: String, minPartitions: Int = defaultMinPartitions): RDD[String] = {
hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
minPartitions).map(pair => pair._2.toString).setName(path)
}
+ def textFile(path: String, minPartitions: Int = defaultMinPartitions,encoding: String = defaultEncoding): RDD[String] = {
hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
minPartitions).map(pair => new String(pair._2.getBytes(), 0 , pair._2.getLength(), encoding)).setName(path)
}
> add a parameter for SparkContext(conf).textFile() method , support for multi-language hdfs file , e.g. "gbk"
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-6316
> URL: https://issues.apache.org/jira/browse/SPARK-6316
> Project: Spark
> Issue Type: New Feature
> Environment: linux
> LANG=en_US.UTF-8
> Reporter: yunzhi.lyz
>
> add a parameter for SparkContext(conf).textFile() method , support for multi-language hdfs file .
>
> e.g. val file = new SparkContext(conf).textFile(args(0), 10,"gbk")
> modify the code：
>
> org.apache.spark.SparkContext
> + def defaultEncoding: String = "utf-8"
>
> -- def textFile(path: String, minPartitions: Int = defaultMinPartitions): RDD[String] = {
> hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
> minPartitions).map(pair => pair._2.toString).setName(path)
> }
> ++ def textFile(path: String, minPartitions: Int = defaultMinPartitions,encoding: String = defaultEncoding): RDD[String] = {
> hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], classOf[Text],
> minPartitions).map(pair => new String(pair._2.getBytes(), 0 , pair._2.getLength(), encoding)).setName(path)
> }
>
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org