You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/08/03 10:42:20 UTC
[jira] [Assigned] (SPARK-16875) Add args checking for DataSet
randomSplit and sample
[ https://issues.apache.org/jira/browse/SPARK-16875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-16875:
------------------------------------
Assignee: (was: Apache Spark)
> Add args checking for DataSet randomSplit and sample
> ----------------------------------------------------
>
> Key: SPARK-16875
> URL: https://issues.apache.org/jira/browse/SPARK-16875
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: zhengruifeng
> Priority: Minor
>
> {code}
> scala> data
> res73: org.apache.spark.sql.DataFrame = [label: double, features: vector]
> scala> data.count
> res74: Long = 150
> scala> val s = data.randomSplit(Array(1,2,-0.01))
> s: Array[org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]] = Array([label: double, features: vector], [label: double, features: vector], [label: double, features: vector])
> scala> s(0).count
> res75: Long = 51
> scala> s(2).count
> 16/08/03 18:28:27 ERROR Executor: Exception in task 0.0 in stage 76.0 (TID 66)
> java.lang.IllegalArgumentException: requirement failed: Upper bound (1.0033444816053512) must be <= 1.0
> at scala.Predef$.require(Predef.scala:224)
> scala> data.sample(false, -0.01)
> res80: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [label: double, features: vector]
> scala> data.sample(false, -0.01).count
> 16/08/03 18:30:33 ERROR Executor: Exception in task 0.0 in stage 84.0 (TID 71)
> java.lang.IllegalArgumentException: requirement failed: Lower bound (0.0) must be <= upper bound (-0.01)
> {code}
> {{val s = data.randomSplit(Array(1,2,-0.01))}} run successfully, even if I use {{s(0)}} in the following lines.
> {{data.sample(false, -0.01)}} should also fail immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org