You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Amareshwari Sriramadasu <am...@apache.org> on 2015/03/20 06:52:31 UTC
Review Request 32281: LENS-404 : Optimize getAll lookups and joinchain
lookups in CubeMetastoreClient
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/
-----------------------------------------------------------
Review request for lens, Jaideep dhok and Rajat Khandelwal.
Bugs: LENS-404
https://issues.apache.org/jira/browse/LENS-404
Repository: lens
Description
-------
Changes include :
- getAll* methods in cube metastore are responded through cache
- isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
- Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
- Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
Diffs
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
Diff: https://reviews.apache.org/r/32281/diff/
Testing
-------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
[INFO] Lens .............................................. SUCCESS [1.769s]
[INFO] Lens API .......................................... SUCCESS [5.111s]
[INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
[INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
[INFO] Lens DB storage ................................... SUCCESS [10.459s]
[INFO] Lens Query Library ................................ SUCCESS [4.952s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
[INFO] Lens Server ....................................... SUCCESS [4:15.372s]
[INFO] Lens client ....................................... SUCCESS [21.028s]
[INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
[INFO] Lens Examples ..................................... SUCCESS [0.769s]
[INFO] Lens Distribution ................................. SUCCESS [9.239s]
[INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
[INFO] Lens Regression ................................... SUCCESS [0.452s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:51.009s
[INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
[INFO] Final Memory: 110M/1132M
[INFO] ------------------------------------------------------------------------
Thanks,
Amareshwari Sriramadasu
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77205
-----------------------------------------------------------
Ship it!
Ship It!
- Rajat Khandelwal
On March 20, 2015, 2:26 p.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 2:26 p.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.130s]
> [INFO] Lens .............................................. SUCCESS [1.841s]
> [INFO] Lens API .......................................... SUCCESS [5.383s]
> [INFO] Lens API for server and extensions ................ SUCCESS [5.997s]
> [INFO] Lens Cube ......................................... SUCCESS [2:03.562s]
> [INFO] Lens DB storage ................................... SUCCESS [10.623s]
> [INFO] Lens Query Library ................................ SUCCESS [4.815s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:34.685s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [18.707s]
> [INFO] Lens Server ....................................... SUCCESS [4:14.833s]
> [INFO] Lens client ....................................... SUCCESS [20.686s]
> [INFO] Lens CLI .......................................... SUCCESS [3:54.751s]
> [INFO] Lens Examples ..................................... SUCCESS [0.862s]
> [INFO] Lens Distribution ................................. SUCCESS [9.474s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.285s]
> [INFO] Lens Regression ................................... SUCCESS [0.479s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 14:52.073s
> [INFO] Finished at: Fri Mar 20 08:52:23 UTC 2015
> [INFO] Final Memory: 111M/1226M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/
-----------------------------------------------------------
(Updated March 20, 2015, 8:56 a.m.)
Review request for lens, Jaideep dhok and Rajat Khandelwal.
Bugs: LENS-404
https://issues.apache.org/jira/browse/LENS-404
Repository: lens
Description
-------
Changes include :
- getAll* methods in cube metastore are responded through cache
- isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
- Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
- Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
Diffs
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
Diff: https://reviews.apache.org/r/32281/diff/
Testing (updated)
-------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.130s]
[INFO] Lens .............................................. SUCCESS [1.841s]
[INFO] Lens API .......................................... SUCCESS [5.383s]
[INFO] Lens API for server and extensions ................ SUCCESS [5.997s]
[INFO] Lens Cube ......................................... SUCCESS [2:03.562s]
[INFO] Lens DB storage ................................... SUCCESS [10.623s]
[INFO] Lens Query Library ................................ SUCCESS [4.815s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:34.685s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [18.707s]
[INFO] Lens Server ....................................... SUCCESS [4:14.833s]
[INFO] Lens client ....................................... SUCCESS [20.686s]
[INFO] Lens CLI .......................................... SUCCESS [3:54.751s]
[INFO] Lens Examples ..................................... SUCCESS [0.862s]
[INFO] Lens Distribution ................................. SUCCESS [9.474s]
[INFO] Lens ML Lib ....................................... SUCCESS [42.285s]
[INFO] Lens Regression ................................... SUCCESS [0.479s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:52.073s
[INFO] Finished at: Fri Mar 20 08:52:23 UTC 2015
[INFO] Final Memory: 111M/1226M
[INFO] ------------------------------------------------------------------------
Thanks,
Amareshwari Sriramadasu
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/
-----------------------------------------------------------
(Updated March 20, 2015, 8:42 a.m.)
Review request for lens, Jaideep dhok and Rajat Khandelwal.
Changes
-------
Trimming for storage table name fixed
Bugs: LENS-404
https://issues.apache.org/jira/browse/LENS-404
Repository: lens
Description
-------
Changes include :
- getAll* methods in cube metastore are responded through cache
- isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
- Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
- Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
Diffs (updated)
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
Diff: https://reviews.apache.org/r/32281/diff/
Testing
-------
Rerunning tests. will update once done
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
[INFO] Lens .............................................. SUCCESS [1.769s]
[INFO] Lens API .......................................... SUCCESS [5.111s]
[INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
[INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
[INFO] Lens DB storage ................................... SUCCESS [10.459s]
[INFO] Lens Query Library ................................ SUCCESS [4.952s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
[INFO] Lens Server ....................................... SUCCESS [4:15.372s]
[INFO] Lens client ....................................... SUCCESS [21.028s]
[INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
[INFO] Lens Examples ..................................... SUCCESS [0.769s]
[INFO] Lens Distribution ................................. SUCCESS [9.239s]
[INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
[INFO] Lens Regression ................................... SUCCESS [0.452s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:51.009s
[INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
[INFO] Final Memory: 110M/1132M
[INFO] ------------------------------------------------------------------------
Thanks,
Amareshwari Sriramadasu
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On March 20, 2015, 7:19 a.m., Rajat Khandelwal wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java, line 420
> > <https://reviews.apache.org/r/32281/diff/1/?file=901168#file901168line420>
> >
> > Why are we doing get after creation everywhere? Lazy cache wasn't working out?
This is mainly to update allCubes. allCubes should start giving newly created cubes as well.
> On March 20, 2015, 7:19 a.m., Rajat Khandelwal wrote:
> > lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java, line 157
> > <https://reviews.apache.org/r/32281/diff/1/?file=901165#file901165line157>
> >
> > Not needed. The contract of `getDimStorages` never allows `null` to be returned. And the case of `isEmpty` is taken care while iterating.
List can be null when there are no storages. Have hit this in tests, so added. isEmpty check during iteration is only for string, not for the list.
> On March 20, 2015, 7:19 a.m., Rajat Khandelwal wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java, line 679
> > <https://reviews.apache.org/r/32281/diff/1/?file=901168#file901168line679>
> >
> > trim needed?
Added with respect all others. But can be removed for storagetablename. Will do.
> On March 20, 2015, 7:19 a.m., Rajat Khandelwal wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1359
> > <https://reviews.apache.org/r/32281/diff/1/?file=901169#file901169line1359>
> >
> > Since `CubeMetastoreClient` accepts both `Table` and `String` as argument for hive tables, it can take both `Cube` and `String` as argument for `getLatestDateOfCube`'s first argument. Then internally, the string version checks for valid cube and calls the cube version.
CubeMetastoreClient does not throw WebApplicationException. Not changing the behavior, as this jira was only doing some improvements.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77175
-----------------------------------------------------------
On March 20, 2015, 7:17 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 7:17 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77175
-----------------------------------------------------------
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java
<https://reviews.apache.org/r/32281/#comment125052>
Not needed. The contract of `getDimStorages` never allows `null` to be returned. And the case of `isEmpty` is taken care while iterating.
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java
<https://reviews.apache.org/r/32281/#comment125055>
Not blocking this jira, Created https://issues.apache.org/jira/browse/LENS-423
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
<https://reviews.apache.org/r/32281/#comment125056>
Why are we doing get after creation everywhere? Lazy cache wasn't working out?
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
<https://reviews.apache.org/r/32281/#comment125057>
trim needed?
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java
<https://reviews.apache.org/r/32281/#comment125058>
Since `CubeMetastoreClient` accepts both `Table` and `String` as argument for hive tables, it can take both `Cube` and `String` as argument for `getLatestDateOfCube`'s first argument. Then internally, the string version checks for valid cube and calls the cube version.
- Rajat Khandelwal
On March 20, 2015, 12:47 p.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 12:47 p.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77188
-----------------------------------------------------------
Ship it!
Ship It!
- Jaideep dhok
On March 20, 2015, 7:17 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 7:17 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Jaideep dhok <ja...@inmobi.com>.
> On March 20, 2015, 7:23 a.m., Jaideep dhok wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java, line 1088
> > <https://reviews.apache.org/r/32281/diff/1-2/?file=901168#file901168line1088>
> >
> > The first get and null check also should be inside the synchronized block, otherwise we risk updating cache twice (two metastore calls)
> >
> > If two threads do get call at the same time, and there's no entry in the cache, then both end up updating the cache, so there's an extra update in this case.
Sorry, not an issue since we are doing containsKey check again.
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77182
-----------------------------------------------------------
On March 20, 2015, 7:17 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 7:17 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77182
-----------------------------------------------------------
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
<https://reviews.apache.org/r/32281/#comment125060>
The first get and null check also should be inside the synchronized block, otherwise we risk updating cache twice (two metastore calls)
If two threads do get call at the same time, and there's no entry in the cache, then both end up updating the cache, so there's an extra update in this case.
- Jaideep dhok
On March 20, 2015, 7:17 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 7:17 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On March 20, 2015, 7:21 a.m., Rajat Khandelwal wrote:
> > So many invocations of `map.get(string.trim().toLowerCase())`. Can make a subclass of `CaseInsensitiveHashMap` and delegate `trim()` calls to there.
Right now, I'm making sure .trim().toLowercase() is called only once per method, as they can start taking time. If we move to subclass of CaseInsensitiveHashMap and do .trim().toLowercase() for every call, we need to evaluate the performance overhead it might bring. We can take it up seperately and evaluate
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77181
-----------------------------------------------------------
On March 20, 2015, 7:17 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 7:17 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77181
-----------------------------------------------------------
So many invocations of `map.get(string.trim().toLowerCase())`. Can make a subclass of `CaseInsensitiveHashMap` and delegate `trim()` calls to there.
- Rajat Khandelwal
On March 20, 2015, 12:47 p.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 12:47 p.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/
-----------------------------------------------------------
(Updated March 20, 2015, 7:17 a.m.)
Review request for lens, Jaideep dhok and Rajat Khandelwal.
Changes
-------
Fixed Locking
Bugs: LENS-404
https://issues.apache.org/jira/browse/LENS-404
Repository: lens
Description
-------
Changes include :
- getAll* methods in cube metastore are responded through cache
- isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
- Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
- Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
Diffs (updated)
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
Diff: https://reviews.apache.org/r/32281/diff/
Testing (updated)
-------
Rerunning tests. will update once done
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
[INFO] Lens .............................................. SUCCESS [1.769s]
[INFO] Lens API .......................................... SUCCESS [5.111s]
[INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
[INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
[INFO] Lens DB storage ................................... SUCCESS [10.459s]
[INFO] Lens Query Library ................................ SUCCESS [4.952s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
[INFO] Lens Server ....................................... SUCCESS [4:15.372s]
[INFO] Lens client ....................................... SUCCESS [21.028s]
[INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
[INFO] Lens Examples ..................................... SUCCESS [0.769s]
[INFO] Lens Distribution ................................. SUCCESS [9.239s]
[INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
[INFO] Lens Regression ................................... SUCCESS [0.452s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:51.009s
[INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
[INFO] Final Memory: 110M/1132M
[INFO] ------------------------------------------------------------------------
Thanks,
Amareshwari Sriramadasu
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Rajat Khandelwal <ra...@gmail.com>.
> On March 20, 2015, 11:38 a.m., Jaideep dhok wrote:
> > Right now none of our cache reads or updates are synchronized. Since we are caching CubeMetastoreClients and these are shared across sessions we should properly synchronize cache calls.
>
> Amareshwari Sriramadasu wrote:
> Will make all the maps ConcurrentHashMap
>
> Jaideep dhok wrote:
> I think just making them ConcurrentHashMap is not enough. All 'get, check and put' transactions also need to be synchronized. For example
>
> ```
> tbl = allHiveTables.get(tableName);
> if (tbl == null) {
> tbl = getClient().getTable(tableName);
> if (enableCaching) {
> allHiveTables.put(tableName, tbl);
> }
> }
> ```
>
> All similar blocks need to be under a synchronized block or guarded by locks.
We can move to guava cache.
- Rajat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77172
-----------------------------------------------------------
On March 20, 2015, 12:47 p.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 12:47 p.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> Rerunning tests. will update once done
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Jaideep dhok <ja...@inmobi.com>.
> On March 20, 2015, 6:08 a.m., Jaideep dhok wrote:
> > Right now none of our cache reads or updates are synchronized. Since we are caching CubeMetastoreClients and these are shared across sessions we should properly synchronize cache calls.
>
> Amareshwari Sriramadasu wrote:
> Will make all the maps ConcurrentHashMap
I think just making them ConcurrentHashMap is not enough. All 'get, check and put' transactions also need to be synchronized. For example
```
tbl = allHiveTables.get(tableName);
if (tbl == null) {
tbl = getClient().getTable(tableName);
if (enableCaching) {
allHiveTables.put(tableName, tbl);
}
}
```
All similar blocks need to be under a synchronized block or guarded by locks.
- Jaideep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77172
-----------------------------------------------------------
On March 20, 2015, 5:52 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 5:52 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On March 20, 2015, 6:08 a.m., Jaideep dhok wrote:
> > Right now none of our cache reads or updates are synchronized. Since we are caching CubeMetastoreClients and these are shared across sessions we should properly synchronize cache calls.
Will make all the maps ConcurrentHashMap
> On March 20, 2015, 6:08 a.m., Jaideep dhok wrote:
> > lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java, line 1533
> > <https://reviews.apache.org/r/32281/diff/1/?file=901168#file901168line1533>
> >
> > Why are all these flags set to enableCaching instead of true. I think setting it to true will make intent explicit.
The cache gets updated only with the flag enableCaching. Otherwise it does not.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77172
-----------------------------------------------------------
On March 20, 2015, 5:52 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 5:52 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>
Re: Review Request 32281: LENS-404 : Optimize getAll lookups and
joinchain lookups in CubeMetastoreClient
Posted by Jaideep dhok <ja...@inmobi.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32281/#review77172
-----------------------------------------------------------
Right now none of our cache reads or updates are synchronized. Since we are caching CubeMetastoreClients and these are shared across sessions we should properly synchronize cache calls.
lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java
<https://reviews.apache.org/r/32281/#comment125050>
Why are all these flags set to enableCaching instead of true. I think setting it to true will make intent explicit.
- Jaideep dhok
On March 20, 2015, 5:52 a.m., Amareshwari Sriramadasu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32281/
> -----------------------------------------------------------
>
> (Updated March 20, 2015, 5:52 a.m.)
>
>
> Review request for lens, Jaideep dhok and Rajat Khandelwal.
>
>
> Bugs: LENS-404
> https://issues.apache.org/jira/browse/LENS-404
>
>
> Repository: lens
>
>
> Description
> -------
>
> Changes include :
> - getAll* methods in cube metastore are responded through cache
> - isCube and isDimension calls changed to look at allCubes and allDimensions instead of hitting metastore when table does not exist
> - Changed the caching keys to have .trim().toLowercase() on all updates/lookups. They are lookups/updates in code which dont trim names, wanted to the cache to agnostic to those.
> - Changed multiple calls of getClient(sessionid) in CubeMetastoreService in single method, as getClient() is not trivial call
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDimensionTableCommands.java 83fb8c8
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeDimensionTable.java a188a41
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeFactTable.java e0e87ab
> lens-cube/src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java 61e511f
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8ea6547
>
> Diff: https://reviews.apache.org/r/32281/diff/
>
>
> Testing
> -------
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.173s]
> [INFO] Lens .............................................. SUCCESS [1.769s]
> [INFO] Lens API .......................................... SUCCESS [5.111s]
> [INFO] Lens API for server and extensions ................ SUCCESS [6.955s]
> [INFO] Lens Cube ......................................... SUCCESS [2:06.767s]
> [INFO] Lens DB storage ................................... SUCCESS [10.459s]
> [INFO] Lens Query Library ................................ SUCCESS [4.952s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:32.339s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [17.878s]
> [INFO] Lens Server ....................................... SUCCESS [4:15.372s]
> [INFO] Lens client ....................................... SUCCESS [21.028s]
> [INFO] Lens CLI .......................................... SUCCESS [2:51.992s]
> [INFO] Lens Examples ..................................... SUCCESS [0.769s]
> [INFO] Lens Distribution ................................. SUCCESS [9.239s]
> [INFO] Lens ML Lib ....................................... SUCCESS [42.785s]
> [INFO] Lens Regression ................................... SUCCESS [0.452s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 13:51.009s
> [INFO] Finished at: Fri Mar 20 05:51:42 UTC 2015
> [INFO] Final Memory: 110M/1132M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Amareshwari Sriramadasu
>
>