[jira] [Resolved] (SPARK-9411) Make page size configurable

["Reynold Xin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Reynold Xin resolved SPARK-9411.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 1.5.0

> Make page size configurable
> ---------------------------
>
>                 Key: SPARK-9411
>                 URL: https://issues.apache.org/jira/browse/SPARK-9411
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Reynold Xin
>            Assignee: Josh Rosen
>            Priority: Critical
>             Fix For: 1.5.0
>
>
> We need to make page sizes configurable so we can reduce them in unit tests and increase them in real production workloads.
> The following hardcoded page sizes need to be updated:
> - Spark Core: UnsafeShuffleExternalSorter.PAGE_SIZE
> - Spark SQL: UnsafeExternalSorter.PAGE_SIZE
> - Unsafe: BytesToBytesMap.PAGE_SIZE_BYTES
> While updating the page sizes, we should also update certain size calculations which are based on the page size so that they do not assume that all pages are the same size.  This isn't strictly necessary in this patch but should be done eventually as part of supporting overflow pages for large records.
> A number of unit tests also need to be updated to account for the new page sizes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org