You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by qinwei <we...@dewmobile.net> on 2014/09/28 08:55:08 UTC

回复: RE: problem with data locality api










Thank you for your reply,
? ? I understand your explaination, but i wonder what is the?correct usage of the apinew              SparkContext(config: SparkConf, preferredNodeLocationData: Map[String, Set[SplitInfo]])how to construct the second param?preferredNodeLocationData?hope for your reply!


qinwei
?发件人：?Shao, Saisai发送时间：?2014-09-28?14:42收件人：?qinwei抄送：?user主题：?RE: problem with data locality api







Hi
?
First conf is used for Hadoop to determine the locality distribution of HDFS file. Second conf is used for Spark, though with the same name, actually
 they are two different classes.
?
Thanks
Jerry
?


From: qinwei [mailto:wei.qin@dewmobile.net]


Sent: Sunday, September 28, 2014 2:05 PM

To: user

Subject: problem with data locality api


?

Hi, everyone


? ? I come across with a problem about data locality, i found these?example?code in
《Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf》


? ??? ??val locData = InputFormatInfo.computePreferredLocations(Seq(new InputFormatInfo(conf,
 classOf[TextInputFormat], new Path(“myfile.txt”)))?

? ??? ??val sc = new SparkContext(conf,
 locData)

? ? but i found the two confs above are of different types, conf in the first line if of type?org.apache.hadoop.conf.Configuration,
 and conf in the second line is of type SparkConf, ?can anyone
 explain that to me or give me some example code?


? ??






qinwei