You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anton Okolnychyi (Jira)" <ji...@apache.org> on 2020/03/27 20:00:03 UTC
[jira] [Comment Edited] (SPARK-23889) DataSourceV2: Add interfaces
to pass required sorting and clustering for writes
[ https://issues.apache.org/jira/browse/SPARK-23889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069018#comment-17069018 ]
Anton Okolnychyi edited comment on SPARK-23889 at 3/27/20, 7:59 PM:
--------------------------------------------------------------------
I've submitted [a proposal|http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-SPARK-23889-DataSourceV2-required-sorting-and-clustering-for-writes-td28985.html] how we can implement this to the dev list. It would be great if people interested in this could comment and we can start working on it.
was (Author: aokolnychyi):
I've submitted [a proposal|[http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-SPARK-23889-DataSourceV2-required-sorting-and-clustering-for-writes-td28985.html]] how we can implement this to the dev list. It would be great if people interested in this could comment and we can start working on it.
> DataSourceV2: Add interfaces to pass required sorting and clustering for writes
> -------------------------------------------------------------------------------
>
> Key: SPARK-23889
> URL: https://issues.apache.org/jira/browse/SPARK-23889
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Ryan Blue
> Priority: Major
>
> From a [discussion on the dev list|https://lists.apache.org/thread.html/d8bb72fc9b4be8acc3f49367bfc99cbf029194a58333eba69df49717@%3Cdev.spark.apache.org%3E], there is consensus around adding interfaces to pass required sorting and clustering to Spark. The proposal is to add:
> {code:java}
> interface RequiresClustering {
> Set<Expression> requiredClustering();
> }
> interface RequiresSort {
> List<SortOrder> requiredOrdering();
> }
> {code}
> When only {{RequiresSort}} is present, the sort would produce a global sort. The partitioning introduced by that sort would be overridden by {{RequiresClustering}}, making the sort local to each partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org