You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by qinwei <we...@dewmobile.net> on 2014/09/28 08:05:05 UTC

problem with data locality api






Hi, everyone? ? I come across with a problem about data locality, i found these?example?code in 《Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf》? ??? ??val locData = InputFormatInfo.computePreferredLocations(Seq(new InputFormatInfo(conf, classOf[TextInputFormat], new Path(“myfile.txt”)))?? ??? ??val sc = new SparkContext(conf, locData)? ? but i found the two confs above are of different types, conf in the first line if of type?org.apache.hadoop.conf.Configuration, and conf in the second line is of type SparkConf, ?can anyone explain that to me or give me some example code?? ??

qinwei

回复: RE: problem with data locality api

Posted by qinwei <we...@dewmobile.net>.









Thank you for your reply,
? ? I understand your explaination, but i wonder what is the?correct usage of the apinew              SparkContext(config: SparkConf, preferredNodeLocationData: Map[String, Set[SplitInfo]])how to construct the second param?preferredNodeLocationData?hope for your reply!


qinwei
?发件人：?Shao, Saisai发送时间：?2014-09-28?14:42收件人：?qinwei抄送：?user主题：?RE: problem with data locality api







Hi
?
First conf is used for Hadoop to determine the locality distribution of HDFS file. Second conf is used for Spark, though with the same name, actually
 they are two different classes.
?
Thanks
Jerry
?


From: qinwei [mailto:wei.qin@dewmobile.net]


Sent: Sunday, September 28, 2014 2:05 PM

To: user

Subject: problem with data locality api


?

Hi, everyone


? ? I come across with a problem about data locality, i found these?example?code in
《Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf》


? ??? ??val locData = InputFormatInfo.computePreferredLocations(Seq(new InputFormatInfo(conf,
 classOf[TextInputFormat], new Path(“myfile.txt”)))?

? ??? ??val sc = new SparkContext(conf,
 locData)

? ? but i found the two confs above are of different types, conf in the first line if of type?org.apache.hadoop.conf.Configuration,
 and conf in the second line is of type SparkConf, ?can anyone
 explain that to me or give me some example code?


? ??






qinwei

RE: problem with data locality api

Posted by "Shao, Saisai" <sa...@intel.com>.
Hi

First conf is used for Hadoop to determine the locality distribution of HDFS file. Second conf is used for Spark, though with the same name, actually they are two different classes.

Thanks
Jerry

From: qinwei [mailto:wei.qin@dewmobile.net]
Sent: Sunday, September 28, 2014 2:05 PM
To: user
Subject: problem with data locality api

Hi, everyone
    I come across with a problem about data locality, i found these example code in 《Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf》
        val locData = InputFormatInfo.computePreferredLocations(Seq(new InputFormatInfo(conf, classOf[TextInputFormat], new Path(“myfile.txt”)))
        val sc = new SparkContext(conf, locData)
    but i found the two confs above are of different types, conf in the first line if of type org.apache.hadoop.conf.Configuration, and conf in the second line is of type SparkConf,  can anyone explain that to me or give me some example code?

________________________________
qinwei