Set the ORC file HDFS block size to 64MB

[Ashish Shenoy <as...@instartlogic.com>]

Hi,

I am using the OrcFile.createWriter() function to get an ORC writer to
write ORC files. However, upon inspecting the ORC files in HDFS, I notice
that they have a block size of 256MB. Now, I would like to set it to 64MB
instead.

I digged around the net and tried a few options:
- I set the "dfs.block.size" config property to 64MB and passed it to the
createWriter() call via OrcFile.writerOptions(config)
- I set the "hive.exec.orc.default.block.size" to 64MB in hive-site.xml
file via ambari and restarted hive.

Neither of these approaches worked for me. I still see that my files have a
block size of 256MB which corresponds to the default setting for the block
size for ORC files. How can I set the block size of ORC files to 64MB ?

Thanks,
Ashish

Re: Set the ORC file HDFS block size to 64MB

[Prasanth Jayachandran <pj...@hortonworks.com>]

Hi

OrcFile.createWriter() methods accepts WriterOptions where you can specify the block size to use using blockSize() method.

Thanks
Prasanth

On Jul 29, 2015, at 5:22 PM, Ashish Shenoy <as...@instartlogic.com>> wrote:

Hi,

I am using the OrcFile.createWriter() function to get an ORC writer to write ORC files. However, upon inspecting the ORC files in HDFS, I notice that they have a block size of 256MB. Now, I would like to set it to 64MB instead.

I digged around the net and tried a few options:
- I set the "dfs.block.size" config property to 64MB and passed it to the createWriter() call via OrcFile.writerOptions(config)
- I set the "hive.exec.orc.default.block.size" to 64MB in hive-site.xml file via ambari and restarted hive.

Neither of these approaches worked for me. I still see that my files have a block size of 256MB which corresponds to the default setting for the block size for ORC files. How can I set the block size of ORC files to 64MB ?

Thanks,
Ashish