[jira] [Comment Edited] (ORC-91) Use hdfs v-blocks instead of zero-padding stripes

["Gopal V (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156107#comment-16156107 ] 

Gopal V edited comment on ORC-91 at 9/6/17 10:15 PM:
-----------------------------------------------------

Found four blocks in a terabyte of inserts which go over the block boundary.

{code}
Stripe: offset: 264747438 data: 29405432 rows: 1085000 tail: 302 index: 23780
{code}

went over the block boundary at 268435456.

Looking into that based on current patch.

(3.5Mb is 5.4% of the stripe size, just below the padding threshold)


was (Author: gopalv):
Found four blocks in a terabyte of inserts which go over the block boundary.

{code}
Stripe: offset: 264747438 data: 29405432 rows: 1085000 tail: 302 index: 23780
{code}

went over the block boundary at 268435456.

Looking into that based on current patch.

> Use hdfs v-blocks instead of zero-padding stripes
> -------------------------------------------------
>
>                 Key: ORC-91
>                 URL: https://issues.apache.org/jira/browse/ORC-91
>             Project: ORC
>          Issue Type: Improvement
>            Reporter: Gopal V
>            Assignee: Gopal V
>
> HDFS-3689 added variable length blocks to HDFS as a core feature.
> hsync(SyncFlag.END_BLOCK) 
> can now end an HDFS block on disk without padding it out with zeros. 
> The current space wasted in ORC to padding is 5%.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)