You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kazuaki Ishizaki (JIRA)" <ji...@apache.org> on 2017/07/12 16:36:01 UTC
[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency
[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084266#comment-16084266 ]
Kazuaki Ishizaki commented on SPARK-21390:
------------------------------------------
Thank you for reporting this. I can reproduce this using Spark 2.2, too.
{code:java}
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.0
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.spark.sql.Dataset
import org.apache.spark.sql.Dataset
scala> case class SomeClass(field1:String, field2:String)
defined class SomeClass
scala> val filterCondition: Seq[SomeClass] = Seq( SomeClass("00", "01") )
filterCondition: Seq[SomeClass] = List(SomeClass(00,01))
scala> val filterMe1: Dataset[SomeClass] = Seq( SomeClass("00", "01") ).toDS
filterMe1: org.apache.spark.sql.Dataset[SomeClass] = [field1: string, field2: string]
scala> println("Works fine!" +filterMe1.filter(filterCondition.contains(_)).count)
Works fine!1
scala> case class OtherClass(field1:String, field2:String)
defined class OtherClass
scala> val filterMe2 = Seq( OtherClass("00", "01"), OtherClass("00", "02")).toDS
filterMe2: org.apache.spark.sql.Dataset[OtherClass] = [field1: string, field2: string]
scala> println("Fail, count should return 1: " + filterMe2.filter(x=> filterCondition.contains(SomeClass(x.field1, x.field2))).count)
Fail, count should return 1: 0
{code}
> Dataset filter api inconsistency
> --------------------------------
>
> Key: SPARK-21390
> URL: https://issues.apache.org/jira/browse/SPARK-21390
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.1
> Reporter: Gheorghe Gheorghe
> Priority: Minor
>
> Hello everybody,
> I've encountered a strange situation with the spark-shell.
> When I run the code below in my IDE the second test case prints as expected count "1". However, when I run the same code using the spark-shell in the second test case I get 0 back as a count.
> I've made sure that I'm running scala 2.11.8 and spark 2.0.1 in both my IDE and spark-shell.
> {code:java}
> import org.apache.spark.sql.Dataset
> case class SomeClass(field1:String, field2:String)
> val filterCondition: Seq[SomeClass] = Seq( SomeClass("00", "01") )
> // Test 1
> val filterMe1: Dataset[SomeClass] = Seq( SomeClass("00", "01") ).toDS
>
> println("Works fine!" +filterMe1.filter(filterCondition.contains(_)).count)
>
> // Test 2
> case class OtherClass(field1:String, field2:String)
>
> val filterMe2 = Seq( OtherClass("00", "01"), OtherClass("00", "02")).toDS
> println("Fail, count should return 1: " + filterMe2.filter(x=> filterCondition.contains(SomeClass(x.field1, x.field2))).count)
> {code}
> Note if I transform the dataset first I get 1 back as expected.
> {code:java}
> println(filterMe2.map(x=> SomeClass(x.field1, x.field2)).filter(filterCondition.contains(_)).count)
> {code}
> Is this a bug? I can see that this filter function has been marked as experimental https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Dataset.html#filter(scala.Function1)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org