Spark's equivalent for Analytical functions in Oracle

[Gireesh Puthumana <gi...@augmentiq.in>]

Hi there,

I would like to check with you whether there is any equivalent functions of
Oracle's analytical functions in Spark SQL.

For example, if I have following data set (table T):
*EID|DEPT*
*101|COMP*
*102|COMP*
*103|COMP*
*104|MARK*

In Oracle, I can do something like
*select EID, DEPT, count(1) over (partition by DEPT) CNT from T;*

to get:
*EID|DEPT|CNT*
*101|COMP|3*
*102|COMP|3*
*103|COMP|3*
*104|MARK|1*

Can we do an equivalent query in Spark-SQL? Or what is the best method to
get such results in Spark dataframes?

Thank you,
Gireesh

Re: Spark's equivalent for Analytical functions in Oracle

[ayan guha <gu...@gmail.com>]

Its available in Spark 1.4 under dataframe window operations. Apparently
programming doc doesnot mention it, you need to look at the apis.

On Mon, Jul 6, 2015 at 8:50 PM, Gireesh Puthumana <
gireesh.puthumana@augmentiq.in> wrote:

> Hi there,
>
> I would like to check with you whether there is any equivalent functions
> of Oracle's analytical functions in Spark SQL.
>
> For example, if I have following data set (table T):
> *EID|DEPT*
> *101|COMP*
> *102|COMP*
> *103|COMP*
> *104|MARK*
>
> In Oracle, I can do something like
> *select EID, DEPT, count(1) over (partition by DEPT) CNT from T;*
>
> to get:
> *EID|DEPT|CNT*
> *101|COMP|3*
> *102|COMP|3*
> *103|COMP|3*
> *104|MARK|1*
>
> Can we do an equivalent query in Spark-SQL? Or what is the best method to
> get such results in Spark dataframes?
>
> Thank you,
> Gireesh
>



-- 
Best Regards,
Ayan Guha