You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2022/03/29 05:55:00 UTC

[jira] [Created] (ORC-1144) [C++] Provide options to trim duplicated values for reader

Quanlong Huang created ORC-1144:
-----------------------------------

             Summary: [C++] Provide options to trim duplicated values for reader
                 Key: ORC-1144
                 URL: https://issues.apache.org/jira/browse/ORC-1144
             Project: ORC
          Issue Type: New Feature
          Components: C++
            Reporter: Quanlong Huang


In case of count-distinct queries, clients just want the distinct values of a column. E.g.
{code:sql}
select count(distinct shipmode) from tpch.lineitem{code}
Column readers can try their best to trim duplicated values in advance. It'd be nice if we have an option/indicator for this purpose.

For dictionary encoded string columns, column readers just need to materailze the dictionary.
For numeric columns that only have PRESENT and DATA streams, the DATA stream decoder can make good use of the encoding types, e.g. SHORT_REPEAT in RLEv2. The PRESENT stream can be skipped as well.

Similar to ORC-1143 and ORC-450, we can extend the ReadIntent to indicate what the users want to do on the results. E.g. adding a ReadIntend_DISTINCT type.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)