You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Gabor Szadovszky (JIRA)" <ji...@apache.org> on 2019/02/19 08:58:00 UTC
[jira] [Commented] (PARQUET-1533) TestSnappy() throws OOM exception
with Parquet-1485 change
[ https://issues.apache.org/jira/browse/PARQUET-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771730#comment-16771730 ]
Gabor Szadovszky commented on PARQUET-1533:
-------------------------------------------
Thanks a lot, [~shangx@uber.com] for creating this JIRA. I haven't realized during the review that we use such large default buffer sizes.
[~liupengcheng], is there a reason behind the default 64M buffer size? (Parquet uses compression on pages which are usually ~1M size.) The original implementation did not use any default size but simply initialized the buffers with 0 and let them grow. Would you like to handle this issue?
> TestSnappy() throws OOM exception with Parquet-1485 change
> -----------------------------------------------------------
>
> Key: PARQUET-1533
> URL: https://issues.apache.org/jira/browse/PARQUET-1533
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.11.0
> Environment: Mac OS 10.14.1
> Reporter: Xinli Shang
> Priority: Minor
>
> Parquet-1485 initialize the buffer size(inputBuffer and outputBuffer) from 0 to 128M in total. This cause the unit test TestSnappy() failed with OOM exception. This is on my Mac laptop.
> To solve the unit test failure, we can increase the size of -Xmx from 512m to 1024m like below. However, we need to evaluate whether or not the increase of the initial size of direct memory usage for inputBuffer and outputBuffer will cause real Parquet application OOM or not, if that application is not with big enough -Xmx size.
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> ...
> <argLine>-Xmx1014m</argLine>
> ...
> For details of the exception, the pull request(https://github.com/apache/parquet-mr/commit/7dcdcdcf0eb5e91618c443d4a84973bf7883d79b) has the detail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)