[jira] [Commented] (ORC-305) Add column statistics for the size on disk

["Owen O'Malley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371701#comment-16371701 ] 

Owen O'Malley commented on ORC-305:
-----------------------------------

So if a stripe has streams like:
{code:java}
{ "id": 0, "column": 0, "kind": "index", "offset": 200000, "length": 7 },
{ "id": 1, "column": 1, "kind": "index", "offset": 200007, "length": 22 },
{ "id": 2, "column": 2, "kind": "index", "offset": 200029, "length": 29 },
{ "id": 3, "column": 3, "kind": "index", "offset": 200058, "length": 39 },
{ "id": 4, "column": 4, "kind": "index", "offset": 200097, "length": 27 },
{ "id": 5, "column": 5, "kind": "index", "offset": 200124, "length": 13 },
{ "id": 6, "column": 1, "kind": "data", "offset": 200137, "length": 54 },
{ "id": 7, "column": 2, "kind": "data", "offset": 200191, "length": 44 },
{ "id": 8, "column": 2, "kind": "length", "offset": 200235, "length": 4 },
{ "id": 9, "column": 2, "kind": "dictionary", "offset": 200239, "length": 8 },
{ "id": 10, "column": 3, "kind": "data", "offset": 200247, "length": 40000 },
{ "id": 11, "column": 4, "kind": "data", "offset": 240247, "length": 5000 },
{ "id": 12, "column": 4, "kind": "secondary", "offset": 245247, "length": 44 },
{ "id": 13, "column": 5, "kind": "data", "offset": 245291, "length": 88 },
{ "id": 14, "column": 5, "kind": "secondary", "offset": 245379, "length": 40 }
{code}

So when updating the column statistics are updated at the stripe, all of the data streams for the column would be added together.

> Add column statistics for the size on disk
> ------------------------------------------
>
>                 Key: ORC-305
>                 URL: https://issues.apache.org/jira/browse/ORC-305
>             Project: ORC
>          Issue Type: Test
>            Reporter: Owen O'Malley
>            Assignee: Sandeep More
>            Priority: Major
>
> It would be great to have the size on disk of each column.
> You can generate this by adding up the sizes of the dictionary and data streams.
> It is only relevant at the stripe and file level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)