[GitHub] [spark] tgravescs commented on a change in pull request #24730: [SPARK-27835][Core] Resource Scheduling: change driver config from addresses

[GitBox <gi...@apache.org>]

tgravescs commented on a change in pull request #24730: [SPARK-27835][Core] Resource Scheduling: change driver config from addresses
URL: https://github.com/apache/spark/pull/24730#discussion_r288575964
 
 

 ##########
 File path: core/src/main/scala/org/apache/spark/ResourceDiscoverer.scala
 ##########
 @@ -132,4 +132,20 @@ private[spark] object ResourceDiscoverer extends Logging {
       }
     }
   }
+
+  def parseAllocatedFromJsonFile(resourcesFile: String): Map[String, ResourceInformation] = {
+    implicit val formats = DefaultFormats
+    // case class to make json4s parsing easy
+    case class JsonResourceInformation(val name: String, val addresses: Array[String])
+    val resourceInput = new BufferedInputStream(new FileInputStream(resourcesFile))
+    val resources = try {
+      parse(resourceInput).extract[Seq[JsonResourceInformation]]
+    } catch {
+      case e@(_: MappingException | _: MismatchedInputException | _: ClassCastException) =>
+        throw new SparkException(s"Exception parsing the resources in $resourcesFile", e)
+    } finally {
+      resourceInput.close()
+    }
+    resources.map(r => (r.name, new ResourceInformation(r.name, r.addresses))).toMap
 
 Review comment:
   Sorry, I'm missing what you are saying here with the toMap.map?  I can't do the toMap until I do the first map to (name, ResourceInformation), otherwise you just have a Seq[JsonResourceInformation] and toMap doesn't know how to make that a map. If there are 2 resource with the same name when the current code runs the toMap will choose the last one.
   
   I had actually tested this and found the json4s parse and extract actually are handling duplicates as well, looks like it chooses the last one, I couldn't find docs on that behavior though either. The resourcesfile is built by the standalone master/worker so it shouldn't really have duplicates.  I'm happy to update though to be explicit so just let me know.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org