You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Vitalii Diravka (JIRA)" <ji...@apache.org> on 2017/05/26 14:45:04 UTC
[jira] [Created] (PARQUET-1006) ColumnChunkPageWriter uses only
heap memory.
Vitalii Diravka created PARQUET-1006:
----------------------------------------
Summary: ColumnChunkPageWriter uses only heap memory.
Key: PARQUET-1006
URL: https://issues.apache.org/jira/browse/PARQUET-1006
Project: Parquet
Issue Type: Bug
Components: parquet-mr
Affects Versions: 1.8.0
Reporter: Vitalii Diravka
Fix For: 1.9.0
After PARQUET-160 was resolved, ColumnChunkPageWriter started using ConcatenatingByteArrayCollector. There are all data is collected in the List of byte[], before writing the page. No way to use direct memory for allocating buffers. ByteBufferAllocator is present in the ColumnChunkPageWriter class, but never used.
Using of java heap space in some cases can cause OOM exceptions or GC's overhead.
ByteBufferAllocator should be used in the ConcatenatingByteArrayCollector or OutputStream classes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)