[jira] [Resolved] (SPARK-9973) Wrong initial size of in-memory columnar buffers

["Cheng Lian (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cheng Lian resolved SPARK-9973.
-------------------------------
    Resolution: Fixed

Resolved by https://github.com/apache/spark/pull/8189

> Wrong initial size of in-memory columnar buffers
> ------------------------------------------------
>
>                 Key: SPARK-9973
>                 URL: https://issues.apache.org/jira/browse/SPARK-9973
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.0
>            Reporter: xukun
>            Assignee: xukun
>             Fix For: 1.5.0
>
>
> Two much memory is allocated for in-memory columnar buffers. The {{initialSize}} argument in {{ColumnBuilder.initialize}} is the initial number of rows rather than bytes, but the value passed in in {{InMemoryColumnarTableScan}} is the latter:
> {code}
> // Class InMemoryColumnarTableScan
>   val initialBufferSize = columnType.defaultSize * batchSize
>   ColumnBuilder(attribute.dataType, initialBufferSize, attribute.name, useCompression)
> {code}
> Then it's converted to byte size again by multiplying {{columnType.defaultSize}}:
> {code}
> // Class BasicColumnBuilder
>   buffer = ByteBuffer.allocate(4 + size * columnType.defaultSize)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org