Re: Review Request 52514: Add data completeness checker

[Narayan Periwal <na...@inmobi.com>]

> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java, line 25
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529514#file1529514line25>
> >
> >     Instead of calling it DefaultCompletenessChecker,please call Mock or something else.

Will rename it as MockCompletenessChecker


> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 939
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line939>
> >
> >     Please add appropriate asserts for the test.

Will add the asserts for the hql query and the expected query.


> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 950
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line950>
> >
> >     Please add appropriate asserts for the test.

Will add the asserts for the hql query and the expected query.


> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 969
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line969>
> >
> >     Not clear whats the expected and actual value here.

Here we are expecting the prune cause to be the Incomplete Data. Hence to assert this, we do a substring of INCOMPLETE_PARTITION.errorFormat


> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/resources/olap-query-conf.xml, line 111
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529512#file1529512line111>
> >
> >     Should we make default to 100% ?

Agreed. We can give default value as 100%.


> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 526
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line526>
> >
> >     This might skip expressions containing measures. We might have to iterate over queriedPhrases and look for measures in them.

Will add the measures after iterating over the queriedPhrases


> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 118
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line118>
> >
> >     Should we make default to 100% ? Also move the default value also as a constant to CubeQueryConfUtil .

Agreed. Will move this to CubeQueryConfUtil.


> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java, line 49
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529508#file1529508line49>
> >
> >     Minor :
> >     
> >     Can we avoid the string concat with " + " ?

Agreed.


- Narayan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review153514
-----------------------------------------------------------


On Oct. 10, 2016, 7:45 a.m., Narayan Periwal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
> 
> (Updated Oct. 10, 2016, 7:45 a.m.)
> 
> 
> Review request for lens.
> 
> 
> Bugs: LENS-1333
>     https://issues.apache.org/jira/browse/LENS-1333
> 
> 
> Repository: lens
> 
> 
> Description
> -------
> 
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
> 
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
> 
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly. 
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
> 
> 
> Diffs
> -----
> 
>   lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3 
>   lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4 
>   lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718 
>   lens-cube/src/main/resources/olap-query-conf.xml 0c888ca 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 6fb027a 
>   lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3 
>   lens-server/src/main/resources/lenssession-default.xml b6bfa2e 
>   src/site/apt/admin/session-config.apt c4b3c04 
>   src/site/apt/user/olap-query-conf.apt ce35ec4 
> 
> Diff: https://reviews.apache.org/r/52514/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Narayan Periwal
> 
>