You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Narayan Periwal <na...@inmobi.com> on 2016/10/04 14:10:15 UTC
Review Request 52514: Add data completeness checker
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs
-----
.gitignore aca87dc
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java, line 25
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529514#file1529514line25>
> >
> > Instead of calling it DefaultCompletenessChecker,please call Mock or something else.
Will rename it as MockCompletenessChecker
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 939
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line939>
> >
> > Please add appropriate asserts for the test.
Will add the asserts for the hql query and the expected query.
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 950
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line950>
> >
> > Please add appropriate asserts for the test.
Will add the asserts for the hql query and the expected query.
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 969
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529517#file1529517line969>
> >
> > Not clear whats the expected and actual value here.
Here we are expecting the prune cause to be the Incomplete Data. Hence to assert this, we do a substring of INCOMPLETE_PARTITION.errorFormat
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/resources/olap-query-conf.xml, line 111
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529512#file1529512line111>
> >
> > Should we make default to 100% ?
Agreed. We can give default value as 100%.
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 526
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line526>
> >
> > This might skip expressions containing measures. We might have to iterate over queriedPhrases and look for measures in them.
Will add the measures after iterating over the queriedPhrases
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 118
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line118>
> >
> > Should we make default to 100% ? Also move the default value also as a constant to CubeQueryConfUtil .
Agreed. Will move this to CubeQueryConfUtil.
> On Oct. 21, 2016, 9:06 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java, line 49
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529508#file1529508line49>
> >
> > Minor :
> >
> > Can we avoid the string concat with " + " ?
Agreed.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review153514
-----------------------------------------------------------
On Oct. 10, 2016, 7:45 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Oct. 10, 2016, 7:45 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
> lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 6fb027a
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-server/src/main/resources/lenssession-default.xml b6bfa2e
> src/site/apt/admin/session-config.apt c4b3c04
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review153514
-----------------------------------------------------------
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java (line 48)
<https://reviews.apache.org/r/52514/#comment222895>
Minor :
Can we avoid the string concat with " + " ?
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java (line 123)
<https://reviews.apache.org/r/52514/#comment222898>
Should pruning cause be added here?
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java (line 125)
<https://reviews.apache.org/r/52514/#comment222896>
incomplete partition's in prune cause is null?
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 112)
<https://reviews.apache.org/r/52514/#comment222900>
Should we make default to 100% ? Also move the default value also as a constant to CubeQueryConfUtil .
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 520)
<https://reviews.apache.org/r/52514/#comment222901>
This might skip expressions containing measures. We might have to iterate over queriedPhrases and look for measures in them.
lens-cube/src/main/resources/olap-query-conf.xml (line 111)
<https://reviews.apache.org/r/52514/#comment222899>
Should we make default to 100% ?
lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java (line 25)
<https://reviews.apache.org/r/52514/#comment222902>
Instead of calling it DefaultCompletenessChecker,please call Mock or something else.
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java (line 939)
<https://reviews.apache.org/r/52514/#comment222903>
Please add appropriate asserts for the test.
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java (line 950)
<https://reviews.apache.org/r/52514/#comment222904>
Please add appropriate asserts for the test.
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java (line 969)
<https://reviews.apache.org/r/52514/#comment222905>
Not clear whats the expected and actual value here.
- Amareshwari Sriramadasu
On Oct. 10, 2016, 7:45 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Oct. 10, 2016, 7:45 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
> lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 6fb027a
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-server/src/main/resources/lenssession-default.xml b6bfa2e
> src/site/apt/admin/session-config.apt c4b3c04
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Oct. 21, 2016, 9:12 a.m., Rajat Khandelwal wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, lines 118-121
> > <https://reviews.apache.org/r/52514/diff/3/?file=1529511#file1529511line118>
> >
> > StorageTableResolver is invoked in each rewrite step, which happens #drivers times for each query. Using reflection to create an instance this often will be harmful for performance.
Agreed. Added this in Hive Conf.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review153515
-----------------------------------------------------------
On Nov. 8, 2016, 1:11 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2016, 1:11 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review153515
-----------------------------------------------------------
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (lines 112 - 115)
<https://reviews.apache.org/r/52514/#comment222906>
StorageTableResolver is invoked in each rewrite step, which happens #drivers times for each query. Using reflection to create an instance this often will be harmful for performance.
- Rajat Khandelwal
On Oct. 10, 2016, 1:15 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Oct. 10, 2016, 1:15 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
> lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 6fb027a
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-server/src/main/resources/lenssession-default.xml b6bfa2e
> src/site/apt/admin/session-config.apt c4b3c04
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Nov. 9, 2016, 11:20 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java, line 155
> > <https://reviews.apache.org/r/52514/diff/5/?file=1556639#file1556639line155>
> >
> > should we add null check for cause.getIncompletePartitions() ?
Yes. I think we should add the null check.
> On Nov. 9, 2016, 11:20 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 35
> > <https://reviews.apache.org/r/52514/diff/5/?file=1556643#file1556643line35>
> >
> > this would result in checkstyle error.
I ran mvn clean install and there was no checkstyle error. Will verify this again.
> On Nov. 9, 2016, 11:20 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 553
> > <https://reviews.apache.org/r/52514/diff/5/?file=1556643#file1556643line553>
> >
> > Why are we breaking after processing one measure? Shouldnt we process all measures in the expression?
This is done to associate the expression with one of the dataCompletenessTag for the measures. So, even if the expression is composed of measures with different dataCompletenessTags, we will be determining the dataCompleteness from one of the measure and this expression is grouped with the other queried measures that have the same dataCompletenessTag.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155418
-----------------------------------------------------------
On Nov. 8, 2016, 1:11 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2016, 1:11 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155418
-----------------------------------------------------------
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java (line 155)
<https://reviews.apache.org/r/52514/#comment225353>
should we add null check for cause.getIncompletePartitions() ?
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 29)
<https://reviews.apache.org/r/52514/#comment225352>
this would result in checkstyle error.
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 547)
<https://reviews.apache.org/r/52514/#comment225359>
Why are we breaking after processing one measure? Shouldnt we process all measures in the expression?
- Amareshwari Sriramadasu
On Nov. 8, 2016, 1:11 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2016, 1:11 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Nov. 10, 2016, 6:01 a.m., Amareshwari Sriramadasu wrote:
> > Overall changes look fine. But I'm just wondering it would be good if we can have a configuration to enable/disable the feature fully - lens.cube.metastore.enable.datacompleteness.checks - if it is set to false, it can fall back to the old code, and not otherwise.
Agreed. It makes sense to add this to enable/disable the feature from this config.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155550
-----------------------------------------------------------
On Nov. 9, 2016, 1:39 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2016, 1:39 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155550
-----------------------------------------------------------
Overall changes look fine. But I'm just wondering it would be good if we can have a configuration to enable/disable the feature fully - lens.cube.metastore.enable.datacompleteness.checks - if it is set to false, it can fall back to the old code, and not otherwise.
- Amareshwari Sriramadasu
On Nov. 9, 2016, 1:39 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2016, 1:39 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Nov. 10, 2016, 7:26 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java, line 99
> > <https://reviews.apache.org/r/52514/diff/7/?file=1560222#file1560222line99>
> >
> > call it isDataCompletenessCheckEnabled or dataCompletenessChecksEnabled ?
> >
> > is and checks in the does not look good.
Sure..Will correct it.
> On Nov. 10, 2016, 7:26 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java, line 49
> > <https://reviews.apache.org/r/52514/diff/7/?file=1560226#file1560226line49>
> >
> > Add entry in olap-query-conf.xml with description.
Agreed.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155554
-----------------------------------------------------------
On Nov. 10, 2016, 7:09 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2016, 7:09 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 8d9f097
> src/site/apt/admin/session-config.apt 663a605
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155554
-----------------------------------------------------------
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java (line 99)
<https://reviews.apache.org/r/52514/#comment225533>
call it isDataCompletenessCheckEnabled or dataCompletenessChecksEnabled ?
is and checks in the does not look good.
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java (line 48)
<https://reviews.apache.org/r/52514/#comment225538>
Add entry in olap-query-conf.xml with description.
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java (line 128)
<https://reviews.apache.org/r/52514/#comment225539>
Lets make the config lens.cube.metastore.completeness.checker.class instead of lens.cube.query.xxx, as it is at server level.
Also, move the config to LensConfConstants and add entry in lensserver-default.xml
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java (lines 131 - 132)
<https://reviews.apache.org/r/52514/#comment225536>
Move these to LensConfConstants and add entry in lensserver-default.xml with description.
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java (line 100)
<https://reviews.apache.org/r/52514/#comment225534>
Can we skip this also if completeness check is disabled ?
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 148)
<https://reviews.apache.org/r/52514/#comment225535>
Should not be under the check.
src/site/apt/admin/session-config.apt (line 105)
<https://reviews.apache.org/r/52514/#comment225537>
Seems there is merge issue. Regenerate the files.
- Amareshwari Sriramadasu
On Nov. 10, 2016, 7:09 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2016, 7:09 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 8d9f097
> src/site/apt/admin/session-config.apt 663a605
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Nov. 10, 2016, 12:02 p.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/resources/olap-query-conf.xml, line 111
> > <https://reviews.apache.org/r/52514/diff/8/?file=1560342#file1560342line111>
> >
> > Please leave default value as blank.
Ok.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155567
-----------------------------------------------------------
On Nov. 10, 2016, 12:06 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2016, 12:06 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f
> lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 6a0230d
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cf1c233
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java PRE-CREATION
> lens-server/src/main/resources/lensserver-default.xml 0ac722d
> lens-server/src/main/resources/lenssession-default.xml 8d9f097
> src/site/apt/admin/config.apt 21b23f9
> src/site/apt/admin/session-config.apt 663a605
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review155567
-----------------------------------------------------------
Fix it, then Ship it!
lens-cube/src/main/resources/olap-query-conf.xml (line 111)
<https://reviews.apache.org/r/52514/#comment225558>
Please leave default value as blank.
- Amareshwari Sriramadasu
On Nov. 10, 2016, 10:39 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 10, 2016, 10:39 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f
> lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 6a0230d
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cf1c233
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java PRE-CREATION
> lens-server/src/main/resources/lensserver-default.xml 0ac722d
> lens-server/src/main/resources/lenssession-default.xml 8d9f097
> src/site/apt/admin/config.apt 21b23f9
> src/site/apt/admin/session-config.apt 663a605
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Nov. 10, 2016, 12:06 p.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 6a0230d
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cf1c233
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java PRE-CREATION
lens-server/src/main/resources/lensserver-default.xml 0ac722d
lens-server/src/main/resources/lenssession-default.xml 8d9f097
src/site/apt/admin/config.apt 21b23f9
src/site/apt/admin/session-config.apt 663a605
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Nov. 10, 2016, 10:39 a.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f
lens-cube/src/main/java/org/apache/lens/cube/parse/DataCompletenessChecker.java 6a0230d
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cf1c233
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DataCompletenessChecker.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/DefaultChecker.java PRE-CREATION
lens-server/src/main/resources/lensserver-default.xml 0ac722d
lens-server/src/main/resources/lenssession-default.xml 8d9f097
src/site/apt/admin/config.apt 21b23f9
src/site/apt/admin/session-config.apt 663a605
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Nov. 10, 2016, 7:09 a.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 408086f
lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
lens-server/src/main/resources/lenssession-default.xml 8d9f097
src/site/apt/admin/session-config.apt 663a605
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Nov. 9, 2016, 1:39 p.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
lens-server/src/main/resources/lenssession-default.xml 9a8f9da
src/site/apt/admin/session-config.apt d480f88
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Nov. 8, 2016, 1:11 p.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
lens-server/src/main/resources/lenssession-default.xml 9a8f9da
src/site/apt/admin/session-config.apt d480f88
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 529
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line529>
> >
> > Measures are getting added, by iterating over queried phrases below. But empty check is happening here. Populating measures should be moved up.
With the empty check for the measures, there is also the empty check of the queriedPhrases with the AND clause. Hence, this check seems to be alright.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 939
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552906#file1552906line939>
> >
> > Why didn't the query fail here ? If the test is to check no failure on partial data, can you update testname and comment above.
Yes, The test is to check no failure on partial data as the flag FAIL_QUERY_ON_PARTIAL_DATA is not set. Will update the test name and add appropriate comment.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java, line 973
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552906#file1552906line973>
> >
> > Can you share the reason being shown up, as a comment?
Sure...will add the comment.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 596
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line596>
> >
> > Shall we rename variable to incompletePartition, instead of partitionCompleteness ?
Will rename it.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, lines 565-566
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line565>
> >
> > Why do you need this formatter? And why is formatting done upto hours?
Actually we can use the Date as well. Agreed, If we use formattter, we should give formatting upto seconds.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review154865
-----------------------------------------------------------
On Nov. 3, 2016, 11:13 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 11:13 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 538
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line538>
> >
> > Can you cross check context.getQueriedMsrs() is actually resolving expression fields as well ? I'm not fully sure.
We will have to use cubeql.getQueriedExprsWithMeasures and parse measures from here.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, line 599
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line599>
> >
> > Instead of measureFromTag should be we put actual queried phrase column here ?
The measures from phrases already gets included in the cubeql.queriedMeasures(). For the expressions, we will put the expression name here instead of the measures that make up the expressions.
> On Nov. 4, 2016, 5:45 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java, lines 603-605
> > <https://reviews.apache.org/r/52514/diff/4/?file=1552900#file1552900line603>
> >
> > Should be moved to outer while loop and should be moved inside the check for isFactDataIncomplete flag for removing there.
Agreed, Will fix it.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review154865
-----------------------------------------------------------
On Nov. 3, 2016, 11:13 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 11:13 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review154865
-----------------------------------------------------------
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 523)
<https://reviews.apache.org/r/52514/#comment224621>
Measures are getting added, by iterating over queried phrases below. But empty check is happening here. Populating measures should be moved up.
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 532)
<https://reviews.apache.org/r/52514/#comment224625>
Can you cross check context.getQueriedMsrs() is actually resolving expression fields as well ? I'm not fully sure.
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (lines 559 - 560)
<https://reviews.apache.org/r/52514/#comment224622>
Why do you need this formatter? And why is formatting done upto hours?
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 590)
<https://reviews.apache.org/r/52514/#comment224623>
Shall we rename variable to incompletePartition, instead of partitionCompleteness ?
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (line 593)
<https://reviews.apache.org/r/52514/#comment224624>
Instead of measureFromTag should be we put actual queried phrase column here ?
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java (lines 597 - 599)
<https://reviews.apache.org/r/52514/#comment224626>
Should be moved to outer while loop and should be moved inside the check for isFactDataIncomplete flag for removing there.
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java (line 939)
<https://reviews.apache.org/r/52514/#comment224627>
Why didn't the query fail here ? If the test is to check no failure on partial data, can you update testname and comment above.
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java (line 973)
<https://reviews.apache.org/r/52514/#comment224628>
Can you share the reason being shown up, as a comment?
- Amareshwari Sriramadasu
On Nov. 3, 2016, 11:13 a.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 11:13 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
> lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
> lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
> lens-server/src/main/resources/lenssession-default.xml 9a8f9da
> src/site/apt/admin/session-config.apt d480f88
> src/site/apt/user/olap-query-conf.apt ce35ec4
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Nov. 3, 2016, 11:13 a.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java e14c43f
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 5dc9dc9
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java f20f105
lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java 0c43cb5
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/MockCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java f6cec1b
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-cube/src/test/java/org/apache/lens/cube/parse/TestQueryRewrite.java 0aa31f4
lens-server/src/main/resources/lenssession-default.xml 9a8f9da
src/site/apt/admin/session-config.apt d480f88
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Oct. 10, 2016, 7:45 a.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/TestBaseCubeQueries.java 6fb027a
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-server/src/main/resources/lenssession-default.xml b6bfa2e
src/site/apt/admin/session-config.apt c4b3c04
src/site/apt/user/olap-query-conf.apt ce35ec4
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/
-----------------------------------------------------------
(Updated Oct. 6, 2016, 10:45 a.m.)
Review request for lens.
Bugs: LENS-1333
https://issues.apache.org/jira/browse/LENS-1333
Repository: lens
Description
-------
Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
Here are some of the capabilities we would like to add in Lens :
# Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
# Lens's accept query on partial data will accept on incomplete data as well.
# Lens will also option to override the completeness percentage threshold value at query level
# Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
# If same measure is there in two different facts , Lens will we pick the one with higher availability.
# In case of completeness percentage threshold missed, Lens will respond back with available percentage.
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
lens-cube/src/main/resources/olap-query-conf.xml 0c888ca
lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
lens-cube/src/test/java/org/apache/lens/cube/parse/TestCubeRewriter.java b90d4d3
lens-server/src/main/resources/lenssession-default.xml b6bfa2e
Diff: https://reviews.apache.org/r/52514/diff/
Testing
-------
Thanks,
Narayan Periwal
Re: Review Request 52514: Add data completeness checker
Posted by Narayan Periwal <na...@inmobi.com>.
> On Oct. 5, 2016, 6:49 a.m., Amareshwari Sriramadasu wrote:
> > .gitignore, line 40
> > <https://reviews.apache.org/r/52514/diff/1/?file=1522015#file1522015line40>
> >
> > Can you avoid this change?
I will revert this change.
> On Oct. 5, 2016, 6:49 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java, line 120
> > <https://reviews.apache.org/r/52514/diff/1/?file=1522019#file1522019line120>
> >
> > Should be added after MISSING_PARTITIONS.
> >
> > Also lets change the text to "Data is incomplete. Details : %s"
Done.
On Oct. 5, 2016, 6:49 a.m., Narayan Periwal wrote:
> > Can you add testcases for all capabilities listed in description?
Adding test cases for all the capabilities listed.
- Narayan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review151465
-----------------------------------------------------------
On Oct. 4, 2016, 2:10 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Oct. 4, 2016, 2:10 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> .gitignore aca87dc
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
> lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>
Re: Review Request 52514: Add data completeness checker
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52514/#review151465
-----------------------------------------------------------
.gitignore (line 40)
<https://reviews.apache.org/r/52514/#comment219818>
Can you avoid this change?
lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java (line 120)
<https://reviews.apache.org/r/52514/#comment219819>
Should be added after MISSING_PARTITIONS.
Also lets change the text to "Data is incomplete. Details : %s"
Can you add testcases for all capabilities listed in description?
- Amareshwari Sriramadasu
On Oct. 4, 2016, 2:10 p.m., Narayan Periwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52514/
> -----------------------------------------------------------
>
> (Updated Oct. 4, 2016, 2:10 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1333
> https://issues.apache.org/jira/browse/LENS-1333
>
>
> Repository: lens
>
>
> Description
> -------
>
> Though lens has partition registration being done whenever data is available, there is no guarantee the partition registered is complete. There can be different ways to know if the data is complete for partition. One option could be to have a partition property saying whether it is complete or not. Other could be to do a http call to another hosted service and more.
>
> Proposal here is to add an interface for DataCompletenessChecker and do the check while resolving partitions.
>
> Here are some of the capabilities we would like to add in Lens :
> # Lens will check partition existence first, if it exists, then check the completeness percentage. If the completeness percentage is less than a configured threshold (default should be 98, 99 or even 100), Lens will fail the query.
> # Lens's accept query on partial data will accept on incomplete data as well.
> # Lens will also option to override the completeness percentage threshold value at query level
> # Lens will still have look ahead capability of daily being incomplete, then it will union with hourly.
> # If same measure is there in two different facts , Lens will we pick the one with higher availability.
> # In case of completeness percentage threshold missed, Lens will respond back with available percentage.
>
>
> Diffs
> -----
>
> .gitignore aca87dc
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java fb958c3
> lens-cube/src/main/java/org/apache/lens/cube/metadata/MetastoreConstants.java 4585ef7
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java 01265a5
> lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTablePruneCause.java 78fb21d
> lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryConfUtil.java 49ed5ef
> lens-cube/src/main/java/org/apache/lens/cube/parse/DefaultChecker.java PRE-CREATION
> lens-cube/src/main/java/org/apache/lens/cube/parse/MaxCoveringFactResolver.java 13f1aa4
> lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java db26718
> lens-cube/src/test/java/org/apache/lens/cube/parse/CubeTestSetup.java f7f8af2
> lens-cube/src/test/java/org/apache/lens/cube/parse/DefaultCompletenessChecker.java PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/cube/parse/FieldsCannotBeQueriedTogetherTest.java 7afa32e
>
> Diff: https://reviews.apache.org/r/52514/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Narayan Periwal
>
>