[jira] [Commented] (SPARK-9411) Make page size configurable

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645295#comment-14645295 ] 

Apache Spark commented on SPARK-9411:
-------------------------------------

User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/7741

> Make page size configurable
> ---------------------------
>
>                 Key: SPARK-9411
>                 URL: https://issues.apache.org/jira/browse/SPARK-9411
>             Project: Spark
>          Issue Type: Improvement
>            Reporter: Reynold Xin
>            Assignee: Josh Rosen
>            Priority: Critical
>
> We need to make page sizes configurable so we can reduce them in unit tests and increase them in real production workloads.
> The following hardcoded page sizes need to be updated:
> - Spark Core: UnsafeShuffleExternalSorter.PAGE_SIZE
> - Spark SQL: UnsafeExternalSorter.PAGE_SIZE
> - Unsafe: BytesToBytesMap.PAGE_SIZE_BYTES
> While updating the page sizes, we should also update certain size calculations which are based on the page size so that they do not assume that all pages are the same size.  This isn't strictly necessary in this patch but should be done eventually as part of supporting overflow pages for large records.
> A number of unit tests also need to be updated to account for the new page sizes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org