You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "qihuagao (JIRA)" <ji...@apache.org> on 2017/07/18 01:55:00 UTC
[jira] [Updated] (SPARK-21448) Hi dear guys, I have a question
about aggregateByKey of pairrrd.
[ https://issues.apache.org/jira/browse/SPARK-21448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
qihuagao updated SPARK-21448:
-----------------------------
Description:
java pair rrd has aggregateByKey, which can avoid full shuffle, so have impressive performance. which has parameters,
The aggregateByKey function requires 3 parameters:
# An intitial ‘zero’ value that will not effect the total values to be collected
# A combining function accepting two paremeters. The second paramter is merged into the first parameter. This function combines/merges values within a partition.
# A merging function function accepting two parameters. In this case the parameters are merged into one. This step merges values across partitions.
While Dataframe, I noticed groupByKey, which could do save function as aggregateByKey, but without merge functions, so I assumed it should trigger shuffle operation. Is this true? if true should we have a funtion like the performance like aggregateByKey for dataframe?
Thanks.
was:
java pair rrd has aggregateByKey, which can avoid full shuffle, so have impressive performance. which has parameters,
The aggregateByKey function requires 3 parameters:
# An intitial ‘zero’ value that will not effect the total values to be collected
# A combining function accepting two paremeters. The second paramter is merged into the first parameter. This function combines/merges values within a partition.
# A merging function function accepting two parameters. In this case the paremters are merged into one. This step merges values across partitions.
While Dataframe, I noticed groupByKey, which could do save function as aggregateByKey, but without merge functions, so I assumed it should trigger shuffle operation. Is this true? if true should we have a funtion like the performance like aggregateByKey for dataframe?
Thanks.
> Hi dear guys, I have a question about aggregateByKey of pairrrd.
> -----------------------------------------------------------------
>
> Key: SPARK-21448
> URL: https://issues.apache.org/jira/browse/SPARK-21448
> Project: Spark
> Issue Type: Question
> Components: Java API
> Affects Versions: 2.0.0
> Environment: Spark 2.0
> Reporter: qihuagao
>
> java pair rrd has aggregateByKey, which can avoid full shuffle, so have impressive performance. which has parameters,
> The aggregateByKey function requires 3 parameters:
> # An intitial ‘zero’ value that will not effect the total values to be collected
> # A combining function accepting two paremeters. The second paramter is merged into the first parameter. This function combines/merges values within a partition.
> # A merging function function accepting two parameters. In this case the parameters are merged into one. This step merges values across partitions.
> While Dataframe, I noticed groupByKey, which could do save function as aggregateByKey, but without merge functions, so I assumed it should trigger shuffle operation. Is this true? if true should we have a funtion like the performance like aggregateByKey for dataframe?
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org