You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marco Gaido (JIRA)" <ji...@apache.org> on 2018/05/04 10:06:00 UTC
[jira] [Commented] (SPARK-24088) only HadoopRDD leverage HDFS Cache
as preferred location
[ https://issues.apache.org/jira/browse/SPARK-24088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463653#comment-16463653 ]
Marco Gaido commented on SPARK-24088:
-------------------------------------
[~xiaojuwu] I don't understand which problem is stated here. {{FileScanRDD}} uses as preferred location the hosts form which the highest number of bytes can be retrieved. What is the problem with this policy? Which issue are you experiencing?
> only HadoopRDD leverage HDFS Cache as preferred location
> --------------------------------------------------------
>
> Key: SPARK-24088
> URL: https://issues.apache.org/jira/browse/SPARK-24088
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 2.3.0
> Reporter: Xiaoju Wu
> Priority: Minor
>
> Only HadoopRDD implements convertSplitLocationInfo which will convert location to HDFSCacheTaskLocation based on if the block is cached in Datanode memory. While FileScanRDD not. In FileScanRDD, all split location information is dropped.
> private[spark] def convertSplitLocationInfo(
> infos: Array[SplitLocationInfo]): Option[Seq[String]] = {
> Option(infos).map(_.flatMap { loc =>
> val locationStr = loc.getLocation
> if (locationStr != "localhost") {
> if (loc.isInMemory) {
> logDebug(s"Partition $locationStr is cached by Hadoop.")
> Some(HDFSCacheTaskLocation(locationStr).toString)
> } else {
> Some(HostTaskLocation(locationStr).toString)
> }
> } else {
> None
> }
> })
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org