[jira] [Created] (HIVE-9573) Lazy load partitions for SELECT LIMIT type query

["Selina Zhang (JIRA)" <ji...@apache.org>]

Selina Zhang created HIVE-9573:
----------------------------------

             Summary: Lazy load partitions for SELECT LIMIT type query
                 Key: HIVE-9573
                 URL: https://issues.apache.org/jira/browse/HIVE-9573
             Project: Hive
          Issue Type: Improvement
          Components: Query Processor
            Reporter: Selina Zhang
            Assignee: Selina Zhang


Some tools such as HUE uses 

SELECT * FROM table LIMIT 100;

to grab the sample content of table. For table with large partitions it causes a large amount of partition objects been loaded. Thus slow down the HS2 even cause OOM from time to time. 

My solution is lazy load partition objects in FetchOperator for this type of query. We can skip retrieving the whole partitions but only partition names in PartitionPrunner, and lazy load the partition objects when needed in execution time (for local job only).

I have a patch ready. But want to hear more suggestions. Thanks!




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)