Re: Review Request 55498: HIVE-15538: Test HIVE-13884 with more complex query predicates

[Chaoyu Tang <ct...@gmail.com>]

> On Jan. 27, 2017, 4:22 p.m., Chaoyu Tang wrote:
> > itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMetaStoreLimitPartitionRequest.java
> > Lines 193 (patched)
> > <https://reviews.apache.org/r/55498/diff/2/?file=1604754#file1604754line193>
> >
> >     How could you tell from the test directly that the query has been falled back to ORM?
> 
> Marta Kuczora wrote:
>     I didn't find a way to explicitly check in the test whether or not the query falls back to ORM. I went through the code and found some rules when the direct SQL cannot be used (for example in the MetaStoreDirectSql.visit method). If the operator LIKE is used or the type of the column/value is invalid or they don't match, direct SQL cannot be used.
>     Also found that if the query falls back to ORM, the number of the partitions can be fetched by the getNumPartitionsViaOrmFilter method or by the getPartitionNamesPrunedByExprNoTxn method. This logic is in the ObjectStore.getNumPartitionsByExpr method:
>     
>             if (exprTree != null) {
>               try {
>                 numPartitions = getNumPartitionsViaOrmFilter(ctx.getTable(), exprTree, true);
>               } catch (MetaException e) {
>                 numPartitions = null;
>               }
>             }
>     
>             // if numPartitions could not be obtained from ORM filters, then get number partitions names, and count them
>             if (numPartitions == null) {
>               List<String> filteredPartNames = new ArrayList<String>();
>               getPartitionNamesPrunedByExprNoTxn(ctx.getTable(), tempExpr, "", (short) -1, filteredPartNames);
>               numPartitions = filteredPartNames.size();
>             }
>     
>     The number of partitions will be fetched by the getNumPartitionsViaOrmFilter method for example for the "select value from %s where num=30 or num=25" query and by the getPartitionNamesPrunedByExprNoTxn method for the "select value from %s where num between 26 and 31" query.
>     I checked these rules and then debugged the queries to see if they really go to the expected way.
>     
>     Do you have any suggestion how I could do this better or how I could check in the test if the query falls back to ORM?

I did not find a better way either :-)


- Chaoyu


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55498/#review163279
-----------------------------------------------------------


On March 31, 2017, 2:52 p.m., Marta Kuczora wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55498/
> -----------------------------------------------------------
> 
> (Updated March 31, 2017, 2:52 p.m.)
> 
> 
> Review request for hive and Chaoyu Tang.
> 
> 
> Bugs: HIVE-15538
>     https://issues.apache.org/jira/browse/HIVE-15538
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Added unit test for testing HIVE-13884 with more complex queries and hive.metastore.limit.partition.request enabled.
> It covers cases when the query predicates can be pushed down and the number of partitions can be retrieved via directSQL.
> It also covers cases when the number of partitions cannot be retrieved via directSQL, so it falls back to ORM.
> 
> 
> Diffs
> -----
> 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMetaStoreLimitPartitionRequest.java PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 80b1e98 
> 
> 
> Diff: https://reviews.apache.org/r/55498/diff/3/
> 
> 
> Testing
> -------
> 
> The patch contains only a new unit test. Ran the test multiple times successfully.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>