You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Jaideep dhok <ja...@inmobi.com> on 2015/03/17 12:31:49 UTC
Review Request 32157: LENS-356 Run estimate for each driver in
parallel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
Review request for lens and Amareshwari Sriramadasu.
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description
-------
This is WIP. Raising review req to get early comments.
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
Known pending items -
1. Config entry for parallel call timeout and its documentation
2. Decide right default value for timeout call.
2. In case estimate call times out for one driver, but the other driver is complete, should we return driver which has completed? Or indicate to user that estimate has failed?
Diffs
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
Diff: https://reviews.apache.org/r/32157/diff/
Testing
-------
Fixed existing impacted tests due to refactor.
Thanks,
Jaideep dhok
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
> On March 19, 2015, 11:06 a.m., Amareshwari Sriramadasu wrote:
> > Looks fine. All tests passed with latest?
Yes. Updated latest test results.
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77044
-----------------------------------------------------------
On March 19, 2015, 10:21 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 10:21 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Updating test results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.458s]
> [INFO] Lens .............................................. SUCCESS [1.856s]
> [INFO] Lens API .......................................... SUCCESS [5.365s]
> [INFO] Lens API for server and extensions ................ SUCCESS [7.854s]
> [INFO] Lens Cube ......................................... SUCCESS [2:03.257s]
> [INFO] Lens DB storage ................................... SUCCESS [10.136s]
> [INFO] Lens Query Library ................................ SUCCESS [4.723s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.905s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.883s]
> [INFO] Lens Server ....................................... SUCCESS [4:17.326s]
> [INFO] Lens client ....................................... SUCCESS [20.524s]
> [INFO] Lens CLI .......................................... SUCCESS [2:44.665s]
> [INFO] Lens Examples ..................................... SUCCESS [0.851s]
> [INFO] Lens Distribution ................................. SUCCESS [10.253s]
> [INFO] Lens ML Lib ....................................... SUCCESS [43.939s]
> [INFO] Lens Regression ................................... SUCCESS [0.448s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:47.419s
> [INFO] Finished at: Thu Mar 19 09:57:17 UTC 2015
> [INFO] Final Memory: 96M/964M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77044
-----------------------------------------------------------
Ship it!
Looks fine. All tests passed with latest?
- Amareshwari Sriramadasu
On March 19, 2015, 10:21 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 10:21 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Updating test results
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.458s]
> [INFO] Lens .............................................. SUCCESS [1.856s]
> [INFO] Lens API .......................................... SUCCESS [5.365s]
> [INFO] Lens API for server and extensions ................ SUCCESS [7.854s]
> [INFO] Lens Cube ......................................... SUCCESS [2:03.257s]
> [INFO] Lens DB storage ................................... SUCCESS [10.136s]
> [INFO] Lens Query Library ................................ SUCCESS [4.723s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.905s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.883s]
> [INFO] Lens Server ....................................... SUCCESS [4:17.326s]
> [INFO] Lens client ....................................... SUCCESS [20.524s]
> [INFO] Lens CLI .......................................... SUCCESS [2:44.665s]
> [INFO] Lens Examples ..................................... SUCCESS [0.851s]
> [INFO] Lens Distribution ................................. SUCCESS [10.253s]
> [INFO] Lens ML Lib ....................................... SUCCESS [43.939s]
> [INFO] Lens Regression ................................... SUCCESS [0.448s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:47.419s
> [INFO] Finished at: Thu Mar 19 09:57:17 UTC 2015
> [INFO] Final Memory: 96M/964M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
(Updated March 19, 2015, 10:21 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Changes
-------
Get estimate call time out from query conf.
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description
-------
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
Diff: https://reviews.apache.org/r/32157/diff/
Testing
-------
Updating test results
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.458s]
[INFO] Lens .............................................. SUCCESS [1.856s]
[INFO] Lens API .......................................... SUCCESS [5.365s]
[INFO] Lens API for server and extensions ................ SUCCESS [7.854s]
[INFO] Lens Cube ......................................... SUCCESS [2:03.257s]
[INFO] Lens DB storage ................................... SUCCESS [10.136s]
[INFO] Lens Query Library ................................ SUCCESS [4.723s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:34.905s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.883s]
[INFO] Lens Server ....................................... SUCCESS [4:17.326s]
[INFO] Lens client ....................................... SUCCESS [20.524s]
[INFO] Lens CLI .......................................... SUCCESS [2:44.665s]
[INFO] Lens Examples ..................................... SUCCESS [0.851s]
[INFO] Lens Distribution ................................. SUCCESS [10.253s]
[INFO] Lens ML Lib ....................................... SUCCESS [43.939s]
[INFO] Lens Regression ................................... SUCCESS [0.448s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:47.419s
[INFO] Finished at: Thu Mar 19 09:57:17 UTC 2015
[INFO] Final Memory: 96M/964M
[INFO] ------------------------------------------------------------------------
Thanks,
Jaideep dhok
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
(Updated March 19, 2015, 10:04 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description
-------
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
Diffs
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
Diff: https://reviews.apache.org/r/32157/diff/
Testing (updated)
-------
Updating test results
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.458s]
[INFO] Lens .............................................. SUCCESS [1.856s]
[INFO] Lens API .......................................... SUCCESS [5.365s]
[INFO] Lens API for server and extensions ................ SUCCESS [7.854s]
[INFO] Lens Cube ......................................... SUCCESS [2:03.257s]
[INFO] Lens DB storage ................................... SUCCESS [10.136s]
[INFO] Lens Query Library ................................ SUCCESS [4.723s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:34.905s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.883s]
[INFO] Lens Server ....................................... SUCCESS [4:17.326s]
[INFO] Lens client ....................................... SUCCESS [20.524s]
[INFO] Lens CLI .......................................... SUCCESS [2:44.665s]
[INFO] Lens Examples ..................................... SUCCESS [0.851s]
[INFO] Lens Distribution ................................. SUCCESS [10.253s]
[INFO] Lens ML Lib ....................................... SUCCESS [43.939s]
[INFO] Lens Regression ................................... SUCCESS [0.448s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:47.419s
[INFO] Finished at: Thu Mar 19 09:57:17 UTC 2015
[INFO] Final Memory: 96M/964M
[INFO] ------------------------------------------------------------------------
Thanks,
Jaideep dhok
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
(Updated March 19, 2015, 9:45 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Changes
-------
Fixed review comments.
Main changes
1. Increased default timeout
2. Fixed gauges
3. Not removing driver from query context
4. Added log handle in AbstractQueryContext and corresponding methods in QueryContext and PreparedQueryContext
5. Cancel driver estimate calls after timeout
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description
-------
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
Diff: https://reviews.apache.org/r/32157/diff/
Testing
-------
Fixed existing impacted tests due to refactor.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
[INFO] Lens .............................................. SUCCESS [1.742s]
[INFO] Lens API .......................................... SUCCESS [5.839s]
[INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
[INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
[INFO] Lens DB storage ................................... SUCCESS [9.854s]
[INFO] Lens Query Library ................................ SUCCESS [5.304s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
[INFO] Lens Server ....................................... SUCCESS [4:21.122s]
[INFO] Lens client ....................................... SUCCESS [20.654s]
[INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
[INFO] Lens Examples ..................................... SUCCESS [0.899s]
[INFO] Lens Distribution ................................. SUCCESS [9.999s]
[INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
[INFO] Lens Regression ................................... SUCCESS [0.453s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:50.381s
[INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
[INFO] Final Memory: 109M/1226M
[INFO] ------------------------------------------------------------------------
Thanks,
Jaideep dhok
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On March 19, 2015, 8:47 a.m., Jaideep dhok wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java, line 1023
> > <https://reviews.apache.org/r/32157/diff/4/?file=899644#file899644line1023>
> >
> > I am not in favor of making it query conf, that way single user can block all jersey threads.
> >
> > Will make it a field variable in query exec service, read at init time.
Just a thought - We can avoid such isuses by making the configuration final in server deployment. But having it at query configuration might allow better testing of the property both for dev and QA
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77029
-----------------------------------------------------------
On March 19, 2015, 6:04 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 6:04 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
> On March 19, 2015, 8:47 a.m., Jaideep dhok wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java, line 1023
> > <https://reviews.apache.org/r/32157/diff/4/?file=899644#file899644line1023>
> >
> > I am not in favor of making it query conf, that way single user can block all jersey threads.
> >
> > Will make it a field variable in query exec service, read at init time.
>
> Amareshwari Sriramadasu wrote:
> Just a thought - We can avoid such isuses by making the configuration final in server deployment. But having it at query configuration might allow better testing of the property both for dev and QA
How can we make it final at deployment time without a code change?
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77029
-----------------------------------------------------------
On March 19, 2015, 9:45 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 9:45 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On March 19, 2015, 8:47 a.m., Jaideep dhok wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java, line 1018
> > <https://reviews.apache.org/r/32157/diff/4/?file=899644#file899644line1018>
> >
> > Changed to what? This is iterator order from ctx.getDriverContext.getDrivers. It is preserved to the end of call.
I meant the order of
estimatePool.submit(r);
runnables.add(r);
But i think i was wrong and it does not matter. runnables are only keeping the ones submitted. Please ignore.
> On March 19, 2015, 8:47 a.m., Jaideep dhok wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java, line 1033
> > <https://reviews.apache.org/r/32157/diff/4/?file=899644#file899644line1033>
> >
> > So that we don't consider it during driver selection.
> >
> > Why do we need the state if its not getting selected anyway?
We are losing the information why the other drivers skipped and the one is picked.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77029
-----------------------------------------------------------
On March 19, 2015, 6:04 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 6:04 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On March 19, 2015, 8:47 a.m., Jaideep dhok wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java, line 1023
> > <https://reviews.apache.org/r/32157/diff/4/?file=899644#file899644line1023>
> >
> > I am not in favor of making it query conf, that way single user can block all jersey threads.
> >
> > Will make it a field variable in query exec service, read at init time.
>
> Amareshwari Sriramadasu wrote:
> Just a thought - We can avoid such isuses by making the configuration final in server deployment. But having it at query configuration might allow better testing of the property both for dev and QA
>
> Jaideep dhok wrote:
> How can we make it final at deployment time without a code change?
Any property can be made final putting the following in lens-site.xml:
<property>
<name>propname</name>
<value>val</value>
<final>true</final>
</property>
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77029
-----------------------------------------------------------
On March 19, 2015, 9:45 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 9:45 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77029
-----------------------------------------------------------
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java
<https://reviews.apache.org/r/32157/#comment124820>
I am not removing the runnable from the map. Only marking it as success or failures. Later during chained exec we will skip estimate if rewrite was failure.
Will add asserts for rewritten query.
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java
<https://reviews.apache.org/r/32157/#comment124821>
This is not thrown at rewrite phase any more since rewrite is not actually happening until later in query exec service.
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
<https://reviews.apache.org/r/32157/#comment124822>
Ok.
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java
<https://reviews.apache.org/r/32157/#comment124823>
Other one sets the whole map. This modifies only for a driver. This is required since each thread will only set query for itself now.
I'll add javadoc.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124824>
Changed to what? This is iterator order from ctx.getDriverContext.getDrivers. It is preserved to the end of call.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124825>
I am not in favor of making it query conf, that way single user can block all jersey threads.
Will make it a field variable in query exec service, read at init time.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124826>
So that we don't consider it during driver selection.
Why do we need the state if its not getting selected anyway?
- Jaideep dhok
On March 19, 2015, 6:04 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 6:04 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
> On March 19, 2015, 9:13 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java, line 1025
> > <https://reviews.apache.org/r/32157/diff/4/?file=899644#file899644line1025>
> >
> > What happens to the estimates that we timed out on waiting, they would continue to run?
> > Do we have to interrupt or stop them?
Will cancel those and add log statement.
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77033
-----------------------------------------------------------
On March 19, 2015, 6:04 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 6:04 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77033
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124831>
What happens to the estimates that we timed out on waiting, they would continue to run?
Do we have to interrupt or stop them?
- Amareshwari Sriramadasu
On March 19, 2015, 6:04 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 6:04 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On March 19, 2015, 6:55 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java, line 305
> > <https://reviews.apache.org/r/32157/diff/4/?file=899638#file899638line305>
> >
> > Not related to this issue, but gauge creation might fail if there are more than one cube query here.
>
> Jaideep dhok wrote:
> How to prevent this?
gauge name has to be unique for each query. Appending the query index would help.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77024
-----------------------------------------------------------
On March 19, 2015, 9:45 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 9:45 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
> On March 19, 2015, 6:55 a.m., Amareshwari Sriramadasu wrote:
> > lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java, line 305
> > <https://reviews.apache.org/r/32157/diff/4/?file=899638#file899638line305>
> >
> > Not related to this issue, but gauge creation might fail if there are more than one cube query here.
How to prevent this?
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77024
-----------------------------------------------------------
On March 19, 2015, 9:45 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 9:45 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java d20c0d09dfee5c2c68875d9f90ed2877675268fa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 202ee1ea423fe363c10663d9e26d09ba1d6712a1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review77024
-----------------------------------------------------------
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java
<https://reviews.apache.org/r/32157/#comment124802>
Not related to this issue, but gauge creation might fail if there are more than one cube query here.
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java
<https://reviews.apache.org/r/32157/#comment124791>
Seems merge issues. The code needs to be put back
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java
<https://reviews.apache.org/r/32157/#comment124795>
I see the number changed here. I think for failing driver rewritten query should not be set. Can we assert that?
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java
<https://reviews.apache.org/r/32157/#comment124793>
Can we assert failure cause contains 'Mock fail' ?
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java
<https://reviews.apache.org/r/32157/#comment124798>
Should be make default value bigger - something in minutes? I'm fine with making it 5 minutes or so to account for bad machines.
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java
<https://reviews.apache.org/r/32157/#comment124803>
typo in the comment
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java
<https://reviews.apache.org/r/32157/#comment124800>
Isnt such method already there?
Can you add javadoc from where it is called?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124813>
Should the order be changed here?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124808>
Is the conf here, server conf? In that case can we read the value only once?
Or should we make configurable per query?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124806>
Why are removing driver from QueryContext? With this we are losing the state fully.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124807>
Estimate can be called on other contexts as well. We can add getLogHandle in AbstractQueryContext which returns user query itself and it can be overriddedn in QueryContext to return queryhandle, PreparedQueryContext to return prepareHandle
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124814>
Can we have a new gauge for this parallel run as a whole?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124809>
Is this still one per driver? I think yes. In that case useStackName should be true.
Also I think the gauge as ALL_CUBE_REWRITES does not make sense any more, can be named differently
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124811>
Please log instead of printing. Whats the behavior when the exception is thrown here? Seems right now it is ignored.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124812>
Same as above
- Amareshwari Sriramadasu
On March 19, 2015, 6:04 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 19, 2015, 6:04 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
(Updated March 19, 2015, 6:04 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Changes
-------
Tune estimate pool and related configs.
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description
-------
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 8da3fd55a95ee8421c85b9d6002a099014497028
lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
Diff: https://reviews.apache.org/r/32157/diff/
Testing
-------
Fixed existing impacted tests due to refactor.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
[INFO] Lens .............................................. SUCCESS [1.742s]
[INFO] Lens API .......................................... SUCCESS [5.839s]
[INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
[INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
[INFO] Lens DB storage ................................... SUCCESS [9.854s]
[INFO] Lens Query Library ................................ SUCCESS [5.304s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
[INFO] Lens Server ....................................... SUCCESS [4:21.122s]
[INFO] Lens client ....................................... SUCCESS [20.654s]
[INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
[INFO] Lens Examples ..................................... SUCCESS [0.899s]
[INFO] Lens Distribution ................................. SUCCESS [9.999s]
[INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
[INFO] Lens Regression ................................... SUCCESS [0.453s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:50.381s
[INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
[INFO] Final Memory: 109M/1226M
[INFO] ------------------------------------------------------------------------
Thanks,
Jaideep dhok
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
(Updated March 18, 2015, 9:43 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Changes
-------
Removed driverQueryContext field.
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description
-------
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
Diff: https://reviews.apache.org/r/32157/diff/
Testing
-------
Fixed existing impacted tests due to refactor.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
[INFO] Lens .............................................. SUCCESS [1.742s]
[INFO] Lens API .......................................... SUCCESS [5.839s]
[INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
[INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
[INFO] Lens DB storage ................................... SUCCESS [9.854s]
[INFO] Lens Query Library ................................ SUCCESS [5.304s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
[INFO] Lens Server ....................................... SUCCESS [4:21.122s]
[INFO] Lens client ....................................... SUCCESS [20.654s]
[INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
[INFO] Lens Examples ..................................... SUCCESS [0.899s]
[INFO] Lens Distribution ................................. SUCCESS [9.999s]
[INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
[INFO] Lens Regression ................................... SUCCESS [0.453s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:50.381s
[INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
[INFO] Final Memory: 109M/1226M
[INFO] ------------------------------------------------------------------------
Thanks,
Jaideep dhok
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review76883
-----------------------------------------------------------
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java
<https://reviews.apache.org/r/32157/#comment124583>
Was seeing some slowness in tests because of contention on query context object monitor between QueryExecService.getStatus call and this call. Since conf update can happen in parallel, using a separate lock for it.
- Jaideep dhok
On March 18, 2015, 8:53 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 18, 2015, 8:53 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
(Updated March 18, 2015, 8:53 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Changes
-------
Updated test result.
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description (updated)
-------
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
Diffs (updated)
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
Diff: https://reviews.apache.org/r/32157/diff/
Testing (updated)
-------
Fixed existing impacted tests due to refactor.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
[INFO] Lens .............................................. SUCCESS [1.742s]
[INFO] Lens API .......................................... SUCCESS [5.839s]
[INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
[INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
[INFO] Lens DB storage ................................... SUCCESS [9.854s]
[INFO] Lens Query Library ................................ SUCCESS [5.304s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
[INFO] Lens Server ....................................... SUCCESS [4:21.122s]
[INFO] Lens client ....................................... SUCCESS [20.654s]
[INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
[INFO] Lens Examples ..................................... SUCCESS [0.899s]
[INFO] Lens Distribution ................................. SUCCESS [9.999s]
[INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
[INFO] Lens Regression ................................... SUCCESS [0.453s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:50.381s
[INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
[INFO] Final Memory: 109M/1226M
[INFO] ------------------------------------------------------------------------
Thanks,
Jaideep dhok
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
On March 18, 2015, 6:46 a.m., Jaideep dhok wrote:
> > Few things:
> >
> > 1. Can we take out common code from the three runnable implementations and make a base class, say `DriverExecutable` ? Generics might be of help.
> > 2. Make an abstract class `DriverExecuter` for executing a collection of `DriverExecutable`s
> > 3. Two implementations of `DriverExecuter`: sequential and parallel. Each have some configurations. e.g. parallel implementation can take some config that specifies whether to run all or run as much as possible in given time frame.
> > 4. Ensure separation of concern between the `DriverExecuter` and its callers.
@Rajat, I agree that some logic to mark success and failure message is repeating in the three classes. However I am not inclined to create a generic ChainRunnable in this patch. That seems like a separate task in itself. If we keep needing it often we can certainly add a util class.
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review76875
-----------------------------------------------------------
On March 18, 2015, 8:53 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 18, 2015, 8:53 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Rajat Khandelwal <ra...@gmail.com>.
On March 18, 2015, 12:16 p.m., Jaideep dhok wrote:
> > Few things:
> >
> > 1. Can we take out common code from the three runnable implementations and make a base class, say `DriverExecutable` ? Generics might be of help.
> > 2. Make an abstract class `DriverExecuter` for executing a collection of `DriverExecutable`s
> > 3. Two implementations of `DriverExecuter`: sequential and parallel. Each have some configurations. e.g. parallel implementation can take some config that specifies whether to run all or run as much as possible in given time frame.
> > 4. Ensure separation of concern between the `DriverExecuter` and its callers.
>
> Jaideep dhok wrote:
> @Rajat, I agree that some logic to mark success and failure message is repeating in the three classes. However I am not inclined to create a generic ChainRunnable in this patch. That seems like a separate task in itself. If we keep needing it often we can certainly add a util class.
Agreed on the point that `ChainDriverExecuter` might be an overkill. But putting down a framework for `DriverExecutable` and `DriverExecuter` might be helpful in later issues. Even if it's not, It will at least make the code more comprehensible. It'll also provide configurability of execution in terms of parallel/sequential execution
- Rajat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review76875
-----------------------------------------------------------
On March 18, 2015, 3:13 p.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 18, 2015, 3:13 p.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
On March 18, 2015, 6:46 a.m., Jaideep dhok wrote:
> > Few things:
> >
> > 1. Can we take out common code from the three runnable implementations and make a base class, say `DriverExecutable` ? Generics might be of help.
> > 2. Make an abstract class `DriverExecuter` for executing a collection of `DriverExecutable`s
> > 3. Two implementations of `DriverExecuter`: sequential and parallel. Each have some configurations. e.g. parallel implementation can take some config that specifies whether to run all or run as much as possible in given time frame.
> > 4. Ensure separation of concern between the `DriverExecuter` and its callers.
>
> Jaideep dhok wrote:
> @Rajat, I agree that some logic to mark success and failure message is repeating in the three classes. However I am not inclined to create a generic ChainRunnable in this patch. That seems like a separate task in itself. If we keep needing it often we can certainly add a util class.
>
> Rajat Khandelwal wrote:
> Agreed on the point that `ChainDriverExecuter` might be an overkill. But putting down a framework for `DriverExecutable` and `DriverExecuter` might be helpful in later issues. Even if it's not, It will at least make the code more comprehensible. It'll also provide configurability of execution in terms of parallel/sequential execution
Can you describe the spec for 'DriverExecutable'
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review76875
-----------------------------------------------------------
On March 18, 2015, 9:43 a.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 18, 2015, 9:43 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
> 5. Proceeding with drivers which return an estimate within a timeout. If no drivers return without an estimate, throwing exception
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java cdc5fd72f63de0f6b4555dd5a08e004013ad7bee
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
> lens-server/src/main/resources/lensserver-default.xml 57fbbc71ede1978d59140f7f3224f90ee8dd2403
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
> src/site/apt/admin/config.apt 9e06da368c27bca97f1cf51f484ecdf101d7caa8
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.093s]
> [INFO] Lens .............................................. SUCCESS [1.742s]
> [INFO] Lens API .......................................... SUCCESS [5.839s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.307s]
> [INFO] Lens Cube ......................................... SUCCESS [2:05.819s]
> [INFO] Lens DB storage ................................... SUCCESS [9.854s]
> [INFO] Lens Query Library ................................ SUCCESS [5.304s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.289s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.355s]
> [INFO] Lens Server ....................................... SUCCESS [4:21.122s]
> [INFO] Lens client ....................................... SUCCESS [20.654s]
> [INFO] Lens CLI .......................................... SUCCESS [1:43.459s]
> [INFO] Lens Examples ..................................... SUCCESS [0.899s]
> [INFO] Lens Distribution ................................. SUCCESS [9.999s]
> [INFO] Lens ML Lib ....................................... SUCCESS [44.152s]
> [INFO] Lens Regression ................................... SUCCESS [0.453s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12:50.381s
> [INFO] Finished at: Wed Mar 18 08:41:20 UTC 2015
> [INFO] Final Memory: 109M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/#review76875
-----------------------------------------------------------
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java
<https://reviews.apache.org/r/32157/#comment124575>
Is this needed? Can this always be figured out using `queryContext` and `driver`?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
<https://reviews.apache.org/r/32157/#comment124576>
Can we abstract some things out to create just a ChainRunnable? Then implement `RewriteEstimateRunnable` using that?
Few things:
1. Can we take out common code from the three runnable implementations and make a base class, say `DriverExecutable` ? Generics might be of help.
2. Make an abstract class `DriverExecuter` for executing a collection of `DriverExecutable`s
3. Two implementations of `DriverExecuter`: sequential and parallel. Each have some configurations. e.g. parallel implementation can take some config that specifies whether to run all or run as much as possible in given time frame.
4. Ensure separation of concern between the `DriverExecuter` and its callers.
- Rajat Khandelwal
On March 17, 2015, 5:03 p.m., Jaideep dhok wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32157/
> -----------------------------------------------------------
>
> (Updated March 17, 2015, 5:03 p.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-356
> https://issues.apache.org/jira/browse/LENS-356
>
>
> Repository: lens
>
>
> Description
> -------
>
> This is WIP. Raising review req to get early comments.
>
> Changes -
> 1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
> 2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
> 3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
> 4. Verified all impacting unit tests they are passing as of now.
>
> Known pending items -
> 1. Config entry for parallel call timeout and its documentation
> 2. Decide right default value for timeout call.
> 2. In case estimate call times out for one driver, but the other driver is complete, should we return driver which has completed? Or indicate to user that estimate has failed? right now I am throwing error in estimate call.
>
>
> Diffs
> -----
>
> lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
> lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
>
> Diff: https://reviews.apache.org/r/32157/diff/
>
>
> Testing
> -------
>
> Fixed existing impacted tests due to refactor.
>
>
> Thanks,
>
> Jaideep dhok
>
>
Re: Review Request 32157: LENS-356 Run estimate for each driver in
parallel
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32157/
-----------------------------------------------------------
(Updated March 17, 2015, 11:33 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Bugs: LENS-356
https://issues.apache.org/jira/browse/LENS-356
Repository: lens
Description (updated)
-------
This is WIP. Raising review req to get early comments.
Changes -
1. Refactored rewrite and estimate calls to return closures instead of directly computing result.
2. QueryExecutionService.rewriteAndSelect will compose a chained runnable for rewrite and estimate and run them in background thread pool
3. Changed existing tests to match current implementation - mainly changed asserts for validation in case of failures. Assertions are still there, just that their verification is changed.
4. Verified all impacting unit tests they are passing as of now.
Known pending items -
1. Config entry for parallel call timeout and its documentation
2. Decide right default value for timeout call.
2. In case estimate call times out for one driver, but the other driver is complete, should we return driver which has completed? Or indicate to user that estimate has failed? right now I am throwing error in estimate call.
Diffs
-----
lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java de79423e701d1fa935d7518fc42d128f261b8b46
lens-cube/src/test/java/org/apache/lens/driver/cube/TestRewriting.java 00a039774b5f9e66db4828800c8f5af2eb9f17b1
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 3101ed6410646f558431781d0833ecc92bde01dc
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java b53c4b982ac55320c27a00e01bd1c5e4170d1263
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 12f6833b70188f72c4c5a9904a6e7d108d7e584c
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java e3bfed817ed988d7e73e802199a1d649f38418be
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 0f20beed578a4aa40b71928199d873e251b81e4e
lens-server/src/test/resources/lens-site.xml 2adf7628d765e34a989e1bc66138d81ca16f949b
Diff: https://reviews.apache.org/r/32157/diff/
Testing
-------
Fixed existing impacted tests due to refactor.
Thanks,
Jaideep dhok