[jira] [Comment Edited] (SPARK-6290) spark.ml.param.Params.checkInputColumn bug upon error

["Glenn Weidner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509538#comment-14509538 ] 

Glenn Weidner edited comment on SPARK-6290 at 4/23/15 6:45 PM:
---------------------------------------------------------------

Thank you Joseph for the quick reply and my apologies for not including test scenario details.  The "java.lang.IllegalArgumentException: requirement failed:" was normal for this case since scala.Predef.require throws the exception when the requirement expression is false.  Note another test scenario was created from org.apache.spark.ml.param.ParamsSuite which resulted in an exception being thrown in getParam method:

  test("SPARK-6290") {

    val field = new StructField("missing", StringType, false)

    val schema = new StructType(Array(field))

    // Force require expression in Params.checkInputColumn to fail since actual is StringType

    solver.checkInputColumn2(schema, "missing", BooleanType)

  }

org.apache.spark.ml.param.TestParams.missing()
java.lang.NoSuchMethodException: org.apache.spark.ml.param.TestParams.missing()
	at java.lang.Class.getMethod(Class.java:1670)
	at org.apache.spark.ml.param.Params$class.getParam(params.scala:163)
	at org.apache.spark.ml.param.TestParams.getParam(TestParams.scala:23)
	at org.apache.spark.ml.param.Params$$anonfun$checkInputColumn$1.apply(params.scala:209)
	at org.apache.spark.ml.param.Params$$anonfun$checkInputColumn$1.apply(params.scala:207)
	at scala.Predef$.require(Predef.scala:233)
	at org.apache.spark.ml.param.Params$class.checkInputColumn(params.scala:206)
	at org.apache.spark.ml.param.TestParams.checkInputColumn(TestParams.scala:23)
	at org.apache.spark.ml.param.TestParams.checkInputColumn2(TestParams.scala:32)
	at org.apache.spark.ml.param.ParamsSuite$$anonfun$5.apply$mcV$sp(ParamsSuite.scala:119)
	at org.apache.spark.ml.param.ParamsSuite$$anonfun$5.apply(ParamsSuite.scala:110)
	at org.apache.spark.ml.param.ParamsSuite$$anonfun$5.apply(ParamsSuite.scala:110)

However, using one of the existing parameters (e.g., "inputCol") defined in TestParams.scala allowed getParam to complete successfully.



was (Author: gweidner):
Thank you Joseph for the quick reply and my apologies for not including test scenario details.  The "java.lang.IllegalArgumentException: requirement failed:" was normal for this case since scala.Predef.require throws the exception when the requirement expression is false.  Note another test scenario was created from org.apache.spark.ml.param.ParamsSuite which resulted in an exception being thrown in getParam method:

  test("SPARK-6290") {
    val field = new StructField("missing", StringType, false)
    val schema = new StructType(Array(field))
    // Force require expression in Params.checkInputColumn to fail since actual is StringType
    solver.checkInputColumn2(schema, "missing", BooleanType)
  }

org.apache.spark.ml.param.TestParams.missing()
java.lang.NoSuchMethodException: org.apache.spark.ml.param.TestParams.missing()
	at java.lang.Class.getMethod(Class.java:1670)
	at org.apache.spark.ml.param.Params$class.getParam(params.scala:163)
	at org.apache.spark.ml.param.TestParams.getParam(TestParams.scala:23)
	at org.apache.spark.ml.param.Params$$anonfun$checkInputColumn$1.apply(params.scala:209)
	at org.apache.spark.ml.param.Params$$anonfun$checkInputColumn$1.apply(params.scala:207)
	at scala.Predef$.require(Predef.scala:233)
	at org.apache.spark.ml.param.Params$class.checkInputColumn(params.scala:206)
	at org.apache.spark.ml.param.TestParams.checkInputColumn(TestParams.scala:23)
	at org.apache.spark.ml.param.TestParams.checkInputColumn2(TestParams.scala:32)
	at org.apache.spark.ml.param.ParamsSuite$$anonfun$5.apply$mcV$sp(ParamsSuite.scala:119)
	at org.apache.spark.ml.param.ParamsSuite$$anonfun$5.apply(ParamsSuite.scala:110)
	at org.apache.spark.ml.param.ParamsSuite$$anonfun$5.apply(ParamsSuite.scala:110)

However, using one of the existing parameters (e.g., "inputCol") defined in TestParams.scala allowed getParam to complete successfully.


> spark.ml.param.Params.checkInputColumn bug upon error
> -----------------------------------------------------
>
>                 Key: SPARK-6290
>                 URL: https://issues.apache.org/jira/browse/SPARK-6290
>             Project: Spark
>          Issue Type: Bug
>          Components: ML
>    Affects Versions: 1.3.0
>            Reporter: Joseph K. Bradley
>            Priority: Minor
>
> In checkInputColumn, if data types do not match, it tries to print an error message with this in it:
> {code}
> Column param description: ${getParam(colName)}"
> {code}
> However, getParam cannot be called on the string colName; it needs the parameter name, which this method is not given.  This causes a weird error which users may find hard to understand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org