You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by anbu <an...@gmail.com> on 2019/03/14 03:55:19 UTC

Windowing LAG function Usage in Spark2.2 Dataset scala

Hi,

To calculate LAG functions difference for the two data_date(current date and
previous date) on the same column
Could you please help me to implement the below scenario using scala spark
Dataset.
uniq_im - LAG(uniq_im,1,0) OVER PARTITION BY(name,sit,plc,country,state)
order by (data_date) as calc_value.

could you please help me how to implement using spark scala Dataset.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Windowing LAG function Usage in Spark2.2 Dataset scala

Posted by Magnus Nilsson <ma...@kth.se>.

import org.apache.spark.sql.expressions.Window

val partitionBy = Window.partitionBy("name", "sit").orderBy("data_date")

val newDf = df.withColumn("PreviousDate", lag("uniq_im",
1).over(partitionBy))

Cheers...

On Thu, Mar 14, 2019 at 4:55 AM anbu <an...@gmail.com> wrote:

> Hi,
>
> To calculate LAG functions difference for the two data_date(current date
> and
> previous date) on the same column
> Could you please help me to implement the below scenario using scala spark
> Dataset.
> uniq_im - LAG(uniq_im,1,0) OVER PARTITION BY(name,sit,plc,country,state)
> order by (data_date) as calc_value.
>
> could you please help me how to implement using spark scala Dataset.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>