You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/02/22 07:31:00 UTC
[jira] [Commented] (SPARK-26962) Windows Function LEAD in Spark SQL
is not fetching consistent results.
[ https://issues.apache.org/jira/browse/SPARK-26962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774865#comment-16774865 ]
Hyukjin Kwon commented on SPARK-26962:
--------------------------------------
Are you able to show the results in the JIRA description? It would be awesome to narrow down and find the root condition that it reads the same data as well.
> Windows Function LEAD in Spark SQL is not fetching consistent results.
> ----------------------------------------------------------------------
>
> Key: SPARK-26962
> URL: https://issues.apache.org/jira/browse/SPARK-26962
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Shiva Sankari Perambalam
> Priority: Major
>
> Using a Lead function on a DATETIME column is giving inconsistent results in Spark sql.
> {code:java}
> Lead(date) over (partition by id, code order by date){code}
> where Date is DATETIME, id and code a String.
> {code:java}
> val testdf1= sparkSession.sql(s""" select date, lead(date) over (partition by id, code order by date) as lead_date from foo"""){code}
> The result set is sometimes having the same data as the date instead of the lead_date
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org