[jira] [Assigned] (SPARK-17663) SchedulableBuilder should handle invalid data access via scheduler.allocation.file

["Imran Rashid (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-17663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Imran Rashid reassigned SPARK-17663:
------------------------------------

    Assignee: Imran Rashid

> SchedulableBuilder should handle invalid data access via scheduler.allocation.file
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-17663
>                 URL: https://issues.apache.org/jira/browse/SPARK-17663
>             Project: Spark
>          Issue Type: Bug
>          Components: Scheduler
>    Affects Versions: 2.1.0
>            Reporter: Eren Avsarogullari
>            Assignee: Imran Rashid
>             Fix For: 2.2.0
>
>
> If spark.scheduler.allocation.file has invalid minShare or/and weight values, these cause :
> - NumberFormatException due to toInt function
> - SparkContext can not be initialized.
> - It does not show meaningful error message to user.
> In a nutshell, this functionality can be more robust by selecting one of the following flows :
> *1-* Currently, if schedulingMode has an invalid value, a warning message is logged and default value is set as FIFO. Same pattern can be used for minShare(default: 0) and weight(default: 1) as well
> *2-* Meaningful error message can be shown to the user for all invalid cases.
> *Code to Reproduce* :
> {code}
> val conf = new SparkConf().setAppName("spark-fairscheduler").setMaster("local")
> conf.set("spark.scheduler.mode", "FAIR")
> conf.set("spark.scheduler.allocation.file", "src/main/resources/fairscheduler-invalid-data.xml")
> val sc = new SparkContext(conf)
> {code}
> *fairscheduler-invalid-data.xml* :
> {code}
> <allocations>
>     <pool name="production">
>         <schedulingMode>FIFO</schedulingMode>
>         <weight>invalid_weight</weight>
>         <minShare>2</minShare>
>     </pool>
> </allocations>
> {code}
> *Stacktrace* :
> {code}
> Exception in thread "main" java.lang.NumberFormatException: For input string: "invalid_weight"
> 	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> 	at java.lang.Integer.parseInt(Integer.java:580)
> 	at java.lang.Integer.parseInt(Integer.java:615)
> 	at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
> 	at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
> 	at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$org$apache$spark$scheduler$FairSchedulableBuilder$$buildFairSchedulerPool$1.apply(SchedulableBuilder.scala:127)
> 	at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$org$apache$spark$scheduler$FairSchedulableBuilder$$buildFairSchedulerPool$1.apply(SchedulableBuilder.scala:102)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org