[jira] [Created] (ORC-74) The ORC writer should suppress the indexes when they are redundant

["Owen O'Malley (JIRA)" <ji...@apache.org>]

Owen O'Malley created ORC-74:
--------------------------------

             Summary: The ORC writer should suppress the indexes when they are redundant
                 Key: ORC-74
                 URL: https://issues.apache.org/jira/browse/ORC-74
             Project: Orc
          Issue Type: Improvement
            Reporter: Owen O'Malley


Currently we always include column indexes and statistics at three levels:
* file
* stripe
* row group (10k rows)

At each level, there isn't any value if the higher level index is exactly the same. Thus, if there is a single stripe in a file, we shouldn't have stripe statistics. If there is a single row group in the stripe, we shouldn't include the row group indexes.

This should improve use cases like Apache NiFi where each file often has less than 10k rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)