You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by luismattor <lu...@gmail.com> on 2016/08/11 00:15:01 UTC

Is there a reduceByKey functionality in DataFrame API?

Hi everyone,

Consider the following code:

val result = df.groupBy("col1").agg(min("col2"))

I know that rdd.reduceByKey(func) produces the same RDD as
rdd.groupByKey().mapValues(value => value.reduce(func)) However reducerByKey
is more efficient as it avoids shipping each value to the reducer doing the
aggregation (it ships partial aggregations instead).

I wonder whether the DataFrame API optimizes the code doing something
similar to what RDD.reduceByKey does. 

I am using Spark 1.6.2. 

Regards,
Luis



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-reduceByKey-functionality-in-DataFrame-API-tp27508.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Re: Is there a reduceByKey functionality in DataFrame API?

Posted by Marius Soutier <mp...@gmail.com>.
Not really, a grouped DataFrame only provides SQL-like functions like sum and avg (at least in 1.5).

> On 29.08.2016, at 14:56, ayan guha <gu...@gmail.com> wrote:
> 
> If you are confused because of  the name of two APIs. I think DF API name groupBy came from SQL, but it works similarly as reducebykey.
> 
> On 29 Aug 2016 20:57, "Marius Soutier" <mps.dev@gmail.com <ma...@gmail.com>> wrote:
> In DataFrames (and thus in 1.5 in general) this is not possible, correct?
> 
>> On 11.08.2016, at 05:42, Holden Karau <holden@pigscanfly.ca <ma...@pigscanfly.ca>> wrote:
>> 
>> Hi Luis,
>> 
>> You might want to consider upgrading to Spark 2.0 - but in Spark 1.6.2 you can do groupBy followed by a reduce on the GroupedDataset ( http://spark.apache.org/docs/1.6.2/api/scala/index.html#org.apache.spark.sql.GroupedDataset <http://spark.apache.org/docs/1.6.2/api/scala/index.html#org.apache.spark.sql.GroupedDataset> ) - this works on a per-key basis despite the different name. In Spark 2.0 you would use groupByKey on the Dataset followed by reduceGroups ( http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.KeyValueGroupedDataset <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.KeyValueGroupedDataset> ).
>> 
>> Cheers,
>> 
>> Holden :)
>> 
>> On Wed, Aug 10, 2016 at 5:15 PM, luismattor <luismattor@gmail.com <ma...@gmail.com>> wrote:
>> Hi everyone,
>> 
>> Consider the following code:
>> 
>> val result = df.groupBy("col1").agg(min("col2"))
>> 
>> I know that rdd.reduceByKey(func) produces the same RDD as
>> rdd.groupByKey().mapValues(value => value.reduce(func)) However reducerByKey
>> is more efficient as it avoids shipping each value to the reducer doing the
>> aggregation (it ships partial aggregations instead).
>> 
>> I wonder whether the DataFrame API optimizes the code doing something
>> similar to what RDD.reduceByKey does.
>> 
>> I am using Spark 1.6.2.
>> 
>> Regards,
>> Luis
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-reduceByKey-functionality-in-DataFrame-API-tp27508.html <http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-reduceByKey-functionality-in-DataFrame-API-tp27508.html>
>> Sent from the Apache Spark User List mailing list archive at Nabble.com <http://nabble.com/>.
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>> 
>> 
>> 
>> 
>> -- 
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau <https://twitter.com/holdenkarau>


Re: Is there a reduceByKey functionality in DataFrame API?

Posted by ayan guha <gu...@gmail.com>.
If you are confused because of  the name of two APIs. I think DF API name
groupBy came from SQL, but it works similarly as reducebykey.
On 29 Aug 2016 20:57, "Marius Soutier" <mp...@gmail.com> wrote:

> In DataFrames (and thus in 1.5 in general) this is not possible, correct?
>
> On 11.08.2016, at 05:42, Holden Karau <ho...@pigscanfly.ca> wrote:
>
> Hi Luis,
>
> You might want to consider upgrading to Spark 2.0 - but in Spark 1.6.2 you
> can do groupBy followed by a reduce on the GroupedDataset (
> http://spark.apache.org/docs/1.6.2/api/scala/index.
> html#org.apache.spark.sql.GroupedDataset ) - this works on a per-key
> basis despite the different name. In Spark 2.0 you would use groupByKey on
> the Dataset followed by reduceGroups ( http://spark.apache.org/
> docs/latest/api/scala/index.html#org.apache.spark.sql.
> KeyValueGroupedDataset ).
>
> Cheers,
>
> Holden :)
>
> On Wed, Aug 10, 2016 at 5:15 PM, luismattor <lu...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Consider the following code:
>>
>> val result = df.groupBy("col1").agg(min("col2"))
>>
>> I know that rdd.reduceByKey(func) produces the same RDD as
>> rdd.groupByKey().mapValues(value => value.reduce(func)) However
>> reducerByKey
>> is more efficient as it avoids shipping each value to the reducer doing
>> the
>> aggregation (it ships partial aggregations instead).
>>
>> I wonder whether the DataFrame API optimizes the code doing something
>> similar to what RDD.reduceByKey does.
>>
>> I am using Spark 1.6.2.
>>
>> Regards,
>> Luis
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/Is-there-a-reduceByKey-functionality-
>> in-DataFrame-API-tp27508.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com
>> <http://nabble.com>.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
>
>
>

Re: Is there a reduceByKey functionality in DataFrame API?

Posted by Marius Soutier <mp...@gmail.com>.
In DataFrames (and thus in 1.5 in general) this is not possible, correct?

> On 11.08.2016, at 05:42, Holden Karau <ho...@pigscanfly.ca> wrote:
> 
> Hi Luis,
> 
> You might want to consider upgrading to Spark 2.0 - but in Spark 1.6.2 you can do groupBy followed by a reduce on the GroupedDataset ( http://spark.apache.org/docs/1.6.2/api/scala/index.html#org.apache.spark.sql.GroupedDataset <http://spark.apache.org/docs/1.6.2/api/scala/index.html#org.apache.spark.sql.GroupedDataset> ) - this works on a per-key basis despite the different name. In Spark 2.0 you would use groupByKey on the Dataset followed by reduceGroups ( http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.KeyValueGroupedDataset <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.KeyValueGroupedDataset> ).
> 
> Cheers,
> 
> Holden :)
> 
> On Wed, Aug 10, 2016 at 5:15 PM, luismattor <luismattor@gmail.com <ma...@gmail.com>> wrote:
> Hi everyone,
> 
> Consider the following code:
> 
> val result = df.groupBy("col1").agg(min("col2"))
> 
> I know that rdd.reduceByKey(func) produces the same RDD as
> rdd.groupByKey().mapValues(value => value.reduce(func)) However reducerByKey
> is more efficient as it avoids shipping each value to the reducer doing the
> aggregation (it ships partial aggregations instead).
> 
> I wonder whether the DataFrame API optimizes the code doing something
> similar to what RDD.reduceByKey does.
> 
> I am using Spark 1.6.2.
> 
> Regards,
> Luis
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-reduceByKey-functionality-in-DataFrame-API-tp27508.html <http://apache-spark-user-list.1001560.n3.nabble.com/Is-there-a-reduceByKey-functionality-in-DataFrame-API-tp27508.html>
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> 
> 
> 
> 
> -- 
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau <https://twitter.com/holdenkarau>

Re: Is there a reduceByKey functionality in DataFrame API?

Posted by Holden Karau <ho...@pigscanfly.ca>.
Hi Luis,

You might want to consider upgrading to Spark 2.0 - but in Spark 1.6.2 you
can do groupBy followed by a reduce on the GroupedDataset (
http://spark.apache.org/docs/1.6.2/api/scala/index.html#org.apache.spark.sql.GroupedDataset
) - this works on a per-key basis despite the different name. In Spark 2.0
you would use groupByKey on the Dataset followed by reduceGroups (
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.KeyValueGroupedDataset
).

Cheers,

Holden :)

On Wed, Aug 10, 2016 at 5:15 PM, luismattor <lu...@gmail.com> wrote:

> Hi everyone,
>
> Consider the following code:
>
> val result = df.groupBy("col1").agg(min("col2"))
>
> I know that rdd.reduceByKey(func) produces the same RDD as
> rdd.groupByKey().mapValues(value => value.reduce(func)) However
> reducerByKey
> is more efficient as it avoids shipping each value to the reducer doing the
> aggregation (it ships partial aggregations instead).
>
> I wonder whether the DataFrame API optimizes the code doing something
> similar to what RDD.reduceByKey does.
>
> I am using Spark 1.6.2.
>
> Regards,
> Luis
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Is-there-a-reduceByKey-functionality-in-DataFrame-
> API-tp27508.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>


-- 
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau