[jira] [Created] (CARBONDATA-4162) Leverage Secondary Index till segment level with Spark plan rewrite

["Nihal kumar ojha (Jira)" <ji...@apache.org>]

Nihal kumar ojha created CARBONDATA-4162:
--------------------------------------------

             Summary: Leverage Secondary Index till segment level with Spark plan rewrite
                 Key: CARBONDATA-4162
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-4162
             Project: CarbonData
          Issue Type: New Feature
            Reporter: Nihal kumar ojha
         Attachments: Support SI at segment level.pdf

*Background:*

Secondary index tables are created as indexes and managed as child tables internally by Carbondata. In the existing architecture, if the parent(main) table and SI table don’t
have the same valid segments then we disable the SI table. And then from the
next query onwards, we scan and prune only the parent table until we trigger
the next load or REINDEX command (as these commands will make the
parent and SI table segments in sync). Because of this, queries take more
time to give the result when SI is disabled.

*Proposed Solution:*
We are planning to leverage SI till the segment level. It means at place
of disabling the SI table(when parent and child table segments are not in sync)
we will do pruning on SI tables for all the valid segments(segments with status
success, marked for update and load partial success) and the rest of the
segments will be pruned by the parent table.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)