You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Alexander Hagerf (Jira)" <ji...@apache.org> on 2019/10/10 10:36:00 UTC
[jira] [Created] (SPARK-29427) Create KeyValueGroupedDataset from
RelationalGroupedDataset
Alexander Hagerf created SPARK-29427:
----------------------------------------
Summary: Create KeyValueGroupedDataset from RelationalGroupedDataset
Key: SPARK-29427
URL: https://issues.apache.org/jira/browse/SPARK-29427
Project: Spark
Issue Type: New Feature
Components: Spark Core, SQL
Affects Versions: 2.4.4
Reporter: Alexander Hagerf
The scenario I'm having is that I'm reading two huge bucketed tables and since a regular join is not performant enough for these cases I'm using groupByKey to generate two KeyValueGroupedDatasets and cogroup them to implement the logic I need.
The issue with this approach is that I'm only grouping by the column that the tables are bucketed by but since I'm using groupByKey the bucketing is completely ignored and I still get a full shuffle.
What I'm looking for is some functionality to tell Catalyst to group by a column in a relational way but then give the user a possibility to utilize the functions of the KeyValueGroupedDataset e.g. cogroup (which is not available for dataframes)
At current spark (2.4.4) I see no way to do this efficiently. I think this is a valid use case which if solved would have huge performance benefits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org