You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Manjunath, Kiran" <ki...@akamai.com> on 2016/10/29 10:28:01 UTC
spark dataframe rolling window for user define operation
Is there a way to get user defined operation to be used for rolling window operation?
Like – Instead of
val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
var dfWithMovingAvg = df.withColumn( "Avg",avg(df("c2")).over(wSpec1))
Something like
val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
var dfWithAlternate = df.withColumn( "alter",XYZ(df("c2")).over(wSpec1))
Where XYZ function can be - +,-,+,- alternatively
PS : I have posted the same question at http://stackoverflow.com/questions/40318010/spark-dataframe-rolling-window-user-define-operation
Regards,
Kiran
Re: spark dataframe rolling window for user define operation
Posted by ayan guha <gu...@gmail.com>.
Avg is an aggregation function. You need to write XYZ as user defined
aggregate function (UDAF).
On Sat, Oct 29, 2016 at 9:28 PM, Manjunath, Kiran <ki...@akamai.com>
wrote:
> Is there a way to get user defined operation to be used for rolling window
> operation?
>
>
>
> Like – Instead of
>
>
>
> val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
>
> var dfWithMovingAvg = df.withColumn( "Avg",avg(df("c2")).over(wSpec1))
>
>
>
> Something like
>
>
>
> val wSpec1 = Window.orderBy("c1").rowsBetween(-20, +20)
>
> var dfWithAlternate = df.withColumn( "alter",*XYZ*(df("c2")).over(wSpec1))
>
>
>
> Where XYZ function can be - +,-,+,- alternatively
>
>
>
>
>
> PS : I have posted the same question at http://stackoverflow.com/
> questions/40318010/spark-dataframe-rolling-window-user-define-operation
>
>
>
> Regards,
>
> Kiran
>
--
Best Regards,
Ayan Guha