You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Suraj Nayak <sn...@gmail.com> on 2014/10/27 19:10:06 UTC
Parquet OOME Java heap space on Large table on Insert Overwrite
Hi Parquet Developers,
Am using Parquet for analytics on AWS EC2. A INSERT OVERWRITE from one
small hive to another tables works fine with parquet with snappy. But when
the file is having 1.2+ Billion records, the insert overwrite fails due to
Java Heap issue.
Few other stats about the data :
Parquet version =1.2.5(CDH5.1.3)
Total files in the HDFS directory : 555
Attached PrintFooter Parquet.txt with the PrintFooter Information. (Note:-
Changes the column names to random characters)
HDFS block size : 256MB
*Yarn configurations :*
yarn.nodemanager.resource.memory-mb = 24GB
yarn.scheduler.minimum-allocation-mb = 2GB
mapreduce.map.memory.mb = 6GB
mapreduce.reduce.memory.mb = 12GB
mapreduce.map.java.opts = 4.5GB
mapreduce.reduce.java.opts = 9GB
yarn.nodemanager.vmem-pmem-ratio = 2.1
Can anyone help me debug or help me fix this issue ? Is this fixed in any
newer Parquet version (>v1.2.5)?
Let me know in case if some one needs more information.
Pasting the task attempt log below :
2014-10-27 13:10:41,556 FATAL [main]
org.apache.hadoop.mapred.YarnChild: Error running child :
java.lang.OutOfMemoryError: Java heap space
at parquet.column.values.dictionary.IntList.initSlab(IntList.java:87)
at parquet.column.values.dictionary.IntList.<init>(IntList.java:83)
at parquet.column.values.dictionary.DictionaryValuesWriter.<init>(DictionaryValuesWriter.java:85)
at parquet.column.values.dictionary.DictionaryValuesWriter$PlainDoubleDictionaryValuesWriter.<init>(DictionaryValuesWriter.java:471)
at parquet.column.impl.ColumnWriterImpl.<init>(ColumnWriterImpl.java:91)
at parquet.column.impl.ColumnWriteStoreImpl.newMemColumn(ColumnWriteStoreImpl.java:60)
at parquet.column.impl.ColumnWriteStoreImpl.getColumnWriter(ColumnWriteStoreImpl.java:52)
at parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.<init>(MessageColumnIO.java:98)
at parquet.io.MessageColumnIO.getRecordWriter(MessageColumnIO.java:283)
at parquet.hadoop.InternalParquetRecordWriter.initStore(InternalParquetRecordWriter.java:102)
at parquet.hadoop.InternalParquetRecordWriter.<init>(InternalParquetRecordWriter.java:88)
at parquet.hadoop.ParquetRecordWriter.<init>(ParquetRecordWriter.java:61)
at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:267)
at parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:245)
at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:54)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:122)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:113)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:296)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:283)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:516)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:692)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:764)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:609)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:847)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:519)
--
Thanks
Suraj Nayak M
Re: Parquet OOME Java heap space on Large table on Insert Overwrite
Posted by Ryan Blue <bl...@cloudera.com>.
On 10/27/2014 12:20 PM, Suraj Nayak wrote:
> On Tue, Oct 28, 2014 at 12:14 AM, Suraj Nayak <snayakm@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Ryan,
>
> Thanks for the detailed info on total memory used.
>
> The output table is partitioned by 2 columns. 1 column has 2 output
> partition and other column has 187 partitions.
>
> parquet.block.size is default. I have not specified it anywhere. If
> you can help me get the exact value, it will be helpful.
>
> Am interested to know the tools for getting this done. Kindly share
> it :)
Suraj,
The parquet.block.size should be 128MB if you've not changed it. You can
always find this value in the configuration properties of your MR job
(or the underlying job in the tracker if you're using Hive).
If you're comfortable writing your own MR job to do this conversion,
then that works. You would just create keys from the data the match the
partition scheme you're using with Hive. Your mapper creates the key for
a record and writes the (key, record) pair. Then the reducer just writes
all of the values it receives.
If you don't want to do this yourself, you can take a look at Kite,
which has this already built so that you can call it from a command-line
interface [1].
rb
[1]:
http://kitesdk.org/docs/current/guide/Using-the-Kite-CLI-to-Create-a-Dataset/
--
Ryan Blue
Software Engineer
Cloudera, Inc.
Re: Parquet OOME Java heap space on Large table on Insert Overwrite
Posted by Suraj Nayak <sn...@gmail.com>.
Is there any way I can write the data from Apache Pig to the above
mentioned partition table via HCatalog?
On Tue, Oct 28, 2014 at 12:14 AM, Suraj Nayak <sn...@gmail.com> wrote:
> Hi Ryan,
>
> Thanks for the detailed info on total memory used.
>
> The output table is partitioned by 2 columns. 1 column has 2 output
> partition and other column has 187 partitions.
>
> parquet.block.size is default. I have not specified it anywhere. If you
> can help me get the exact value, it will be helpful.
>
> Am interested to know the tools for getting this done. Kindly share it :)
>
>
>
> On Mon, Oct 27, 2014 at 11:53 PM, Ryan Blue <bl...@cloudera.com> wrote:
>
>> On 10/27/2014 11:10 AM, Suraj Nayak wrote:
>>
>>> Hi Parquet Developers,
>>>
>>> Am using Parquet for analytics on AWS EC2. A INSERT OVERWRITE from one
>>> small hive to another tables works fine with parquet with snappy. But
>>> when the file is having 1.2+ Billion records, the insert overwrite fails
>>> due to Java Heap issue.
>>>
>>> Few other stats about the data :
>>>
>>> Parquet version =1.2.5(CDH5.1.3)
>>> Total files in the HDFS directory : 555
>>> Attached PrintFooter Parquet.txt with the PrintFooter Information.
>>> (Note:- Changes the column names to random characters)
>>> HDFS block size : 256MB
>>>
>>> *Yarn configurations :*
>>> yarn.nodemanager.resource.memory-mb = 24GB
>>> yarn.scheduler.minimum-allocation-mb = 2GB
>>> mapreduce.map.memory.mb = 6GB
>>> mapreduce.reduce.memory.mb = 12GB
>>> mapreduce.map.java.opts = 4.5GB
>>> mapreduce.reduce.java.opts = 9GB
>>> yarn.nodemanager.vmem-pmem-ratio = 2.1
>>>
>>>
>>> Can anyone help me debug or help me fix this issue ? Is this fixed in
>>> any newer Parquet version (>v1.2.5)?
>>>
>>> Let me know in case if some one needs more information.
>>>
>>
>> Hi Suraj,
>>
>> Can you give us a little more info? Specifically:
>> 1. Approximately how many output partitions are you writing to?
>> 2. What is the parquet.block.size setting in your config?
>>
>> It looks like you're probably running into trouble with the number of
>> open files, so I'll assume that . . .
>>
>> Hive will open a file in each partition written to by a task, which will
>> use up to parquet.block.size bytes of memory (or about that). So the total
>> memory used is parquet.block.size * # partitions. You can fix the problem
>> by reducing the number of output partitions, which usually requires that
>> you break up the conversion into smaller jobs. You can also reduce the
>> parquet block size, but I don't recommend it.
>>
>> Another option is to use another parquet library for the conversion, and
>> shuffle the data in a MR round so that each reducer writes to just one
>> partition. If you need help doing this, I can recommend some tools.
>>
>> rb
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Cloudera, Inc.
>>
>
>
>
> --
> Thanks
> Suraj Nayak M
>
--
Thanks
Suraj Nayak M
Re: Parquet OOME Java heap space on Large table on Insert Overwrite
Posted by Suraj Nayak <sn...@gmail.com>.
Hi Ryan,
Thanks for the detailed info on total memory used.
The output table is partitioned by 2 columns. 1 column has 2 output
partition and other column has 187 partitions.
parquet.block.size is default. I have not specified it anywhere. If you can
help me get the exact value, it will be helpful.
Am interested to know the tools for getting this done. Kindly share it :)
On Mon, Oct 27, 2014 at 11:53 PM, Ryan Blue <bl...@cloudera.com> wrote:
> On 10/27/2014 11:10 AM, Suraj Nayak wrote:
>
>> Hi Parquet Developers,
>>
>> Am using Parquet for analytics on AWS EC2. A INSERT OVERWRITE from one
>> small hive to another tables works fine with parquet with snappy. But
>> when the file is having 1.2+ Billion records, the insert overwrite fails
>> due to Java Heap issue.
>>
>> Few other stats about the data :
>>
>> Parquet version =1.2.5(CDH5.1.3)
>> Total files in the HDFS directory : 555
>> Attached PrintFooter Parquet.txt with the PrintFooter Information.
>> (Note:- Changes the column names to random characters)
>> HDFS block size : 256MB
>>
>> *Yarn configurations :*
>> yarn.nodemanager.resource.memory-mb = 24GB
>> yarn.scheduler.minimum-allocation-mb = 2GB
>> mapreduce.map.memory.mb = 6GB
>> mapreduce.reduce.memory.mb = 12GB
>> mapreduce.map.java.opts = 4.5GB
>> mapreduce.reduce.java.opts = 9GB
>> yarn.nodemanager.vmem-pmem-ratio = 2.1
>>
>>
>> Can anyone help me debug or help me fix this issue ? Is this fixed in
>> any newer Parquet version (>v1.2.5)?
>>
>> Let me know in case if some one needs more information.
>>
>
> Hi Suraj,
>
> Can you give us a little more info? Specifically:
> 1. Approximately how many output partitions are you writing to?
> 2. What is the parquet.block.size setting in your config?
>
> It looks like you're probably running into trouble with the number of open
> files, so I'll assume that . . .
>
> Hive will open a file in each partition written to by a task, which will
> use up to parquet.block.size bytes of memory (or about that). So the total
> memory used is parquet.block.size * # partitions. You can fix the problem
> by reducing the number of output partitions, which usually requires that
> you break up the conversion into smaller jobs. You can also reduce the
> parquet block size, but I don't recommend it.
>
> Another option is to use another parquet library for the conversion, and
> shuffle the data in a MR round so that each reducer writes to just one
> partition. If you need help doing this, I can recommend some tools.
>
> rb
>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>
--
Thanks
Suraj Nayak M
Re: Parquet OOME Java heap space on Large table on Insert Overwrite
Posted by Ryan Blue <bl...@cloudera.com>.
On 10/27/2014 11:10 AM, Suraj Nayak wrote:
> Hi Parquet Developers,
>
> Am using Parquet for analytics on AWS EC2. A INSERT OVERWRITE from one
> small hive to another tables works fine with parquet with snappy. But
> when the file is having 1.2+ Billion records, the insert overwrite fails
> due to Java Heap issue.
>
> Few other stats about the data :
>
> Parquet version =1.2.5(CDH5.1.3)
> Total files in the HDFS directory : 555
> Attached PrintFooter Parquet.txt with the PrintFooter Information.
> (Note:- Changes the column names to random characters)
> HDFS block size : 256MB
>
> *Yarn configurations :*
> yarn.nodemanager.resource.memory-mb = 24GB
> yarn.scheduler.minimum-allocation-mb = 2GB
> mapreduce.map.memory.mb = 6GB
> mapreduce.reduce.memory.mb = 12GB
> mapreduce.map.java.opts = 4.5GB
> mapreduce.reduce.java.opts = 9GB
> yarn.nodemanager.vmem-pmem-ratio = 2.1
>
>
> Can anyone help me debug or help me fix this issue ? Is this fixed in
> any newer Parquet version (>v1.2.5)?
>
> Let me know in case if some one needs more information.
Hi Suraj,
Can you give us a little more info? Specifically:
1. Approximately how many output partitions are you writing to?
2. What is the parquet.block.size setting in your config?
It looks like you're probably running into trouble with the number of
open files, so I'll assume that . . .
Hive will open a file in each partition written to by a task, which will
use up to parquet.block.size bytes of memory (or about that). So the
total memory used is parquet.block.size * # partitions. You can fix the
problem by reducing the number of output partitions, which usually
requires that you break up the conversion into smaller jobs. You can
also reduce the parquet block size, but I don't recommend it.
Another option is to use another parquet library for the conversion, and
shuffle the data in a MR round so that each reducer writes to just one
partition. If you need help doing this, I can recommend some tools.
rb
--
Ryan Blue
Software Engineer
Cloudera, Inc.