[jira] [Commented] (SPARK-21581) Spark 2.x distinct return incorrect result

["shengyao piao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108324#comment-16108324 ] 

shengyao piao commented on SPARK-21581:
---------------------------------------

[~maropu]
Very appreciated for your detailed explanation.
I understood about this behavior.

Thank you!


> Spark 2.x distinct return incorrect result
> ------------------------------------------
>
>                 Key: SPARK-21581
>                 URL: https://issues.apache.org/jira/browse/SPARK-21581
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.0.0, 2.1.0, 2.2.0
>            Reporter: shengyao piao
>
> Hi all
> I'm using Spark2.x on cdh5.11
> I have a json file as follows.
> ・sample.json
> {code}
> {"url": "http://example.hoge/staff1", "name": "staff1", "salary":600.0}
> {"url": "http://example.hoge/staff2", "name": "staff2", "salary":700}
> {"url": "http://example.hoge/staff3", "name": "staff3", "salary":800}
> {"url": "http://example.hoge/staff4", "name": "staff4", "salary":900}
> {"url": "http://example.hoge/staff5", "name": "staff5", "salary":1000.0}
> {"url": "http://example.hoge/staff6", "name": "staff6", "salary":""}
> {"url": "http://example.hoge/staff7", "name": "staff7", "salary":""}
> {"url": "http://example.hoge/staff8", "name": "staff8", "salary":""}
> {"url": "http://example.hoge/staff9", "name": "staff9", "salary":""}
> {"url": "http://example.hoge/staff10", "name": "staff10", "salary":""}
> {code}
> And I try to read this file and distinct.
> ・spark code
> {code}
> val s=spark.read.json("sample.json")
> s.count
> res13: Long = 10
> s.distinct.count
> res14: Long = 6    < - It's should be 10
> {code}
> I know the cause of incorrect result is by mixed type in salary field.
> But when I try the same code in Spark 1.6 the result will be 10.
> So I think it's a bug in Spark 2.x.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org