You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/06/19 09:25:00 UTC
[jira] [Commented] (SPARK-28094) Multiple left joins or
aggregations in one query produce incorrect results
[ https://issues.apache.org/jira/browse/SPARK-28094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867431#comment-16867431 ]
Jungtaek Lim commented on SPARK-28094:
--------------------------------------
NOTE: SPARK-28074 is derived on the investigation of this issue when this was asked from user mailing list before.
> Multiple left joins or aggregations in one query produce incorrect results
> --------------------------------------------------------------------------
>
> Key: SPARK-28094
> URL: https://issues.apache.org/jira/browse/SPARK-28094
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.3
> Reporter: Joe Ammann
> Priority: Major
>
> Structured streaming queries which involve more than one non-map-like transformations (either left joins or aggregations) produce incorrect (and sometimes inconsistent) results.
> I have put up a sample here as Gist https://gist.github.com/jammann/b58bfbe0f4374b89ecea63c1e32c8f17
> The test case involves 4 topics, and I've implement 6 different use cases of which 3 fail to produce the expected results
> 1) A innerjoin B: to topic A_B
> expected: 10 inner joined messages
> observation: OK
> 2) A innerjoin B outerjoin C: to topic A_B_outer_C
> expected: 9 inner joined messages, last one in watermark
> observation: OK
> 3) A innerjoin B outerjoin C outerjoin D: to topic A_B_outer_C_outer_D
> expected: 9 inner/outer joined messages, 3 match C, 1 match D, last one in watermark
> observation: NOK, only 3 messages are produced, where the C outer join matches
> 4) B outerjoin C: to topic B_outer_C
> expected: 9 outer joined messages, 3 match C, 1 match D, last one in watermark
> observation: NOK, same as 3
> 5) A innerjoin B agg on field of A: to topic A_inner_B_agg
> expected: 3 groups with a total of 9 inner joined messages, last one in watermark
> observation: OK
> 6) A innerjoin B outerjoin C agg on field of A: to topic A_inner_B_outer_C_agg
> expected: 3 groups with a total of 9 inner joined messages, last one in watermark
> observation: NOK, 3 groups with 1 message each, where C outer join matches
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org