You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Amareshwari Sriramadasu <am...@apache.org> on 2016/08/04 08:57:45 UTC
Review Request 50783: LENS-1251 : Fix adding resources on HiveDriver
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50783/
-----------------------------------------------------------
Review request for lens.
Bugs: LENS-1251
https://issues.apache.org/jira/browse/LENS-1251
Repository: lens
Description
-------
The bug is because of LensSessionImpl.getDBResources() is returning failedDBResources instead of actual resources.
Changes include :
- Removed failedDBResources from LensSessionImpl - which is not solving the purpose. Once resource addition has failed, it would fail in subsequent tries as well.
- Instead of adding resouces to HiveDriver on every execute/explain/prepare and etc, adding resources on DriverSessionStarted event - synchronously. This will make sure resources are added whenever new session is created and the classloader will be uptodate in all cases. For ex: Query on a native table fails in estimate with earlier code, because jars are not added and HiveDriver will do explain on the query.
- Book-keeping in LensSessionImpl is only used in tests and all source code usages are removed.
- Removed book-keeping in HiveDriver which is maintaining whether resources are added for a session or not.
Diffs
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 1326611
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 84dcecd
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 3003ab7
lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 20aa8cb
lens-server/src/test/java/org/apache/lens/server/metastore/TestMetastoreService.java 61bc133
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 9f40a01
Diff: https://reviews.apache.org/r/50783/diff/
Testing
-------
Added regression test which fails with ClassNotFoundException without the changes.
Thanks,
Amareshwari Sriramadasu
Re: Review Request 50783: LENS-1251 : Fix adding resources on
HiveDriver
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50783/#review146496
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java (line 3142)
<https://reviews.apache.org/r/50783/#comment212974>
Doing addition of session resources only upon new session would miss the resources that user adds later through commandline.
- Amareshwari Sriramadasu
On Aug. 4, 2016, 10:22 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50783/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2016, 10:22 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1251
> https://issues.apache.org/jira/browse/LENS-1251
>
>
> Repository: lens
>
>
> Description
> -------
>
> The bug is because of LensSessionImpl.getDBResources() is returning failedDBResources instead of actual resources.
>
> Changes include :
> - Removed failedDBResources from LensSessionImpl - which is not solving the purpose. Once resource addition has failed, it would fail in subsequent tries as well.
> - Instead of adding resouces to HiveDriver on every execute/explain/prepare and etc, adding resources on DriverSessionStarted event - synchronously. This will make sure resources are added whenever new session is created and the classloader will be uptodate in all cases. For ex: Query on a native table fails in estimate with earlier code, because jars are not added and HiveDriver will do explain on the query.
> - Book-keeping in LensSessionImpl is only used in tests and all source code usages are removed.
> - Removed book-keeping in HiveDriver which is maintaining whether resources are added for a session or not.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 1326611
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 84dcecd
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 3003ab7
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 20aa8cb
> lens-server/src/test/java/org/apache/lens/server/metastore/TestMetastoreService.java 61bc133
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 9f40a01
>
> Diff: https://reviews.apache.org/r/50783/diff/
>
>
> Testing
> -------
>
> Added regression test which fails with ClassNotFoundException without the changes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [1.608s]
> [INFO] Lens .............................................. SUCCESS [3.837s]
> [INFO] Lens API .......................................... SUCCESS [21.476s]
> [INFO] Lens API for server and extensions ................ SUCCESS [18.842s]
> [INFO] Lens Cube ......................................... SUCCESS [15:44.025s]
> [INFO] Lens DB storage ................................... SUCCESS [16.969s]
> [INFO] Lens Query Library ................................ SUCCESS [14.158s]
> [INFO] Lens Hive Driver .................................. SUCCESS [1:22.657s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [36.501s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [21.397s]
> [INFO] Lens Server ....................................... SUCCESS [9:14.088s]
> [INFO] Lens client ....................................... SUCCESS [1:37.241s]
> [INFO] Lens CLI .......................................... SUCCESS [1:29.427s]
> [INFO] Lens Examples ..................................... SUCCESS [8.243s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.618s]
> [INFO] Lens Distribution ................................. SUCCESS [9.133s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:15.242s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [2.808s]
> [INFO] Lens Regression ................................... SUCCESS [16.170s]
> [INFO] Lens UI ........................................... SUCCESS [33.806s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 34:08.950s
> [INFO] Finished at: Thu Aug 04 10:14:00 UTC 2016
> [INFO] Final Memory: 185M/2352M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 50783: LENS-1251 : Fix adding resources on
HiveDriver
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50783/#review145303
-----------------------------------------------------------
+1 for the changes. They are not only fixing the issue but also simplifying the code.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java (line 3123)
<https://reviews.apache.org/r/50783/#comment211481>
Will new jars added to session need separate handling ?
- Puneet Gupta
On Aug. 4, 2016, 10:22 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50783/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2016, 10:22 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1251
> https://issues.apache.org/jira/browse/LENS-1251
>
>
> Repository: lens
>
>
> Description
> -------
>
> The bug is because of LensSessionImpl.getDBResources() is returning failedDBResources instead of actual resources.
>
> Changes include :
> - Removed failedDBResources from LensSessionImpl - which is not solving the purpose. Once resource addition has failed, it would fail in subsequent tries as well.
> - Instead of adding resouces to HiveDriver on every execute/explain/prepare and etc, adding resources on DriverSessionStarted event - synchronously. This will make sure resources are added whenever new session is created and the classloader will be uptodate in all cases. For ex: Query on a native table fails in estimate with earlier code, because jars are not added and HiveDriver will do explain on the query.
> - Book-keeping in LensSessionImpl is only used in tests and all source code usages are removed.
> - Removed book-keeping in HiveDriver which is maintaining whether resources are added for a session or not.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 1326611
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 84dcecd
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 3003ab7
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 20aa8cb
> lens-server/src/test/java/org/apache/lens/server/metastore/TestMetastoreService.java 61bc133
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 9f40a01
>
> Diff: https://reviews.apache.org/r/50783/diff/
>
>
> Testing
> -------
>
> Added regression test which fails with ClassNotFoundException without the changes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [1.608s]
> [INFO] Lens .............................................. SUCCESS [3.837s]
> [INFO] Lens API .......................................... SUCCESS [21.476s]
> [INFO] Lens API for server and extensions ................ SUCCESS [18.842s]
> [INFO] Lens Cube ......................................... SUCCESS [15:44.025s]
> [INFO] Lens DB storage ................................... SUCCESS [16.969s]
> [INFO] Lens Query Library ................................ SUCCESS [14.158s]
> [INFO] Lens Hive Driver .................................. SUCCESS [1:22.657s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [36.501s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [21.397s]
> [INFO] Lens Server ....................................... SUCCESS [9:14.088s]
> [INFO] Lens client ....................................... SUCCESS [1:37.241s]
> [INFO] Lens CLI .......................................... SUCCESS [1:29.427s]
> [INFO] Lens Examples ..................................... SUCCESS [8.243s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.618s]
> [INFO] Lens Distribution ................................. SUCCESS [9.133s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:15.242s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [2.808s]
> [INFO] Lens Regression ................................... SUCCESS [16.170s]
> [INFO] Lens UI ........................................... SUCCESS [33.806s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 34:08.950s
> [INFO] Finished at: Thu Aug 04 10:14:00 UTC 2016
> [INFO] Final Memory: 185M/2352M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 50783: LENS-1251 : Fix adding resources on
HiveDriver
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50783/
-----------------------------------------------------------
(Updated Aug. 4, 2016, 10:22 a.m.)
Review request for lens.
Bugs: LENS-1251
https://issues.apache.org/jira/browse/LENS-1251
Repository: lens
Description
-------
The bug is because of LensSessionImpl.getDBResources() is returning failedDBResources instead of actual resources.
Changes include :
- Removed failedDBResources from LensSessionImpl - which is not solving the purpose. Once resource addition has failed, it would fail in subsequent tries as well.
- Instead of adding resouces to HiveDriver on every execute/explain/prepare and etc, adding resources on DriverSessionStarted event - synchronously. This will make sure resources are added whenever new session is created and the classloader will be uptodate in all cases. For ex: Query on a native table fails in estimate with earlier code, because jars are not added and HiveDriver will do explain on the query.
- Book-keeping in LensSessionImpl is only used in tests and all source code usages are removed.
- Removed book-keeping in HiveDriver which is maintaining whether resources are added for a session or not.
Diffs
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 1326611
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 84dcecd
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 3003ab7
lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 20aa8cb
lens-server/src/test/java/org/apache/lens/server/metastore/TestMetastoreService.java 61bc133
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java 9f40a01
Diff: https://reviews.apache.org/r/50783/diff/
Testing (updated)
-------
Added regression test which fails with ClassNotFoundException without the changes.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [1.608s]
[INFO] Lens .............................................. SUCCESS [3.837s]
[INFO] Lens API .......................................... SUCCESS [21.476s]
[INFO] Lens API for server and extensions ................ SUCCESS [18.842s]
[INFO] Lens Cube ......................................... SUCCESS [15:44.025s]
[INFO] Lens DB storage ................................... SUCCESS [16.969s]
[INFO] Lens Query Library ................................ SUCCESS [14.158s]
[INFO] Lens Hive Driver .................................. SUCCESS [1:22.657s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [36.501s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [21.397s]
[INFO] Lens Server ....................................... SUCCESS [9:14.088s]
[INFO] Lens client ....................................... SUCCESS [1:37.241s]
[INFO] Lens CLI .......................................... SUCCESS [1:29.427s]
[INFO] Lens Examples ..................................... SUCCESS [8.243s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.618s]
[INFO] Lens Distribution ................................. SUCCESS [9.133s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:15.242s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [2.808s]
[INFO] Lens Regression ................................... SUCCESS [16.170s]
[INFO] Lens UI ........................................... SUCCESS [33.806s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 34:08.950s
[INFO] Finished at: Thu Aug 04 10:14:00 UTC 2016
[INFO] Final Memory: 185M/2352M
[INFO] ------------------------------------------------------------------------
Thanks,
Amareshwari Sriramadasu