You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/07/24 10:57:00 UTC
[jira] [Assigned] (SPARK-24900) speed up sort when the dataset is
small
[ https://issues.apache.org/jira/browse/SPARK-24900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-24900:
------------------------------------
Assignee: Apache Spark
> speed up sort when the dataset is small
> ---------------------------------------
>
> Key: SPARK-24900
> URL: https://issues.apache.org/jira/browse/SPARK-24900
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: SongXun
> Assignee: Apache Spark
> Priority: Minor
> Labels: pull-request-available
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> when running the sql like 'select * from order where order_status = 4 order by order_id', the pysical plan is:
> !https://ws1.sinaimg.cn/large/006tNc79ly1ftl01gtso1j31kw0ag43q.jpg!
> the Exchange rangepartitioning has two steps:
> 1. sample the rdd and get the RangePartitioner which has a rangeBounds
> !https://ws1.sinaimg.cn/large/006tNc79ly1ftl0a6ziytj30le0su0vw.jpg!
> 2. get the rddWithPartitionIds depending on the rangeBounds, and do the shuffle
> !https://ws1.sinaimg.cn/large/006tNc79ly1ftl0bapn6mj30l40ke769.jpg!
> !https://ws1.sinaimg.cn/large/006tNc79ly1ftl0brnj3jj30kq0mu40q.jpg!
> The filescan and filter will be executed twice, it may take a long time. If the final dataset is small, and the sample data covers all the data, there is no need to do so.
> !https://ws3.sinaimg.cn/large/006tNc79ly1ftl3u0091wj31600a8wh1.jpg!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org