You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Deepak Barr <de...@gmail.com> on 2015/09/29 20:59:44 UTC
Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/
-----------------------------------------------------------
Review request for lens.
Repository: lens
Description
-------
Currently, To obtain list of native tables, CubeMetastoreService does the following -
1. Fetches the list of tables ( one MetastoreClient call)
2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
Diffs
-----
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
Diff: https://reviews.apache.org/r/38857/diff/
Testing
-------
Yes.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
[INFO] Lens ............................................... SUCCESS [ 4.601 s]
[INFO] Lens API ........................................... SUCCESS [ 31.397 s]
[INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
[INFO] Lens Cube .......................................... SUCCESS [07:04 min]
[INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
[INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
[INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
[INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
[INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
[INFO] Lens Server ........................................ SUCCESS [11:11 min]
[INFO] Lens client ........................................ SUCCESS [ 48.547 s]
[INFO] Lens CLI ........................................... SUCCESS [03:45 min]
[INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
[INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
[INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
[INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
[INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
[INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:53 min
[INFO] Finished at: 2015-09-29T22:48:22+05:30
[INFO] Final Memory: 175M/3158M
[INFO] ------------------------------------------------------------------------
Thanks,
Deepak Barr
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Raju Bairishetti <ra...@gmail.com>.
> On Sept. 30, 2015, 1:37 a.m., Raju Bairishetti wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1203
> > <https://reviews.apache.org/r/38857/diff/1/?file=1087309#file1087309line1203>
> >
> > Why are we changing exceptions here?
> >
> > You may need to InvalidOperationExcpetion if you keep the older method declaration (i.e. throws MetaException, UnknownDBException, HiveSQLException, TException, LensException).
> >
> >
> > MetaException, UnknownDBException and InvalidOperationException all extends TException.
>
> Deepak Barr wrote:
> Raju,
>
> I didnt quite get you. Since TException extends MetaException, UnknownDBException, I felt we can instead throw a general TException. I dint get your point about InvalidOperationExcpetion though.
I would say keep the existing exceptions(MetaException, UnknownException). getTableObjectsByName(...) is throwing InvalidOperationException also. We may need to throw InvalidOperationException.
It would be easier to identify the problem from the actual exception instead of TException.
++ Rajat
- Raju
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101061
-----------------------------------------------------------
On Sept. 29, 2015, 6:59 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Sept. 29, 2015, 6:59 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Deepak Barr <de...@gmail.com>.
> On Sept. 30, 2015, 1:37 a.m., Raju Bairishetti wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1203
> > <https://reviews.apache.org/r/38857/diff/1/?file=1087309#file1087309line1203>
> >
> > Why are we changing exceptions here?
> >
> > You may need to InvalidOperationExcpetion if you keep the older method declaration (i.e. throws MetaException, UnknownDBException, HiveSQLException, TException, LensException).
> >
> >
> > MetaException, UnknownDBException and InvalidOperationException all extends TException.
>
> Deepak Barr wrote:
> Raju,
>
> I didnt quite get you. Since TException extends MetaException, UnknownDBException, I felt we can instead throw a general TException. I dint get your point about InvalidOperationExcpetion though.
>
> Raju Bairishetti wrote:
> I would say keep the existing exceptions(MetaException, UnknownException). getTableObjectsByName(...) is throwing InvalidOperationException also. We may need to throw InvalidOperationException.
>
> It would be easier to identify the problem from the actual exception instead of TException.
>
> ++ Rajat
Ohhh ok. I got your point. Thanks !
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101061
-----------------------------------------------------------
On Sept. 29, 2015, 6:59 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Sept. 29, 2015, 6:59 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Deepak Barr <de...@gmail.com>.
> On Sept. 30, 2015, 1:37 a.m., Raju Bairishetti wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1203
> > <https://reviews.apache.org/r/38857/diff/1/?file=1087309#file1087309line1203>
> >
> > Why are we changing exceptions here?
> >
> > You may need to InvalidOperationExcpetion if you keep the older method declaration (i.e. throws MetaException, UnknownDBException, HiveSQLException, TException, LensException).
> >
> >
> > MetaException, UnknownDBException and InvalidOperationException all extends TException.
Raju,
I didnt quite get you. Since TException extends MetaException, UnknownDBException, I felt we can instead throw a general TException. I dint get your point about InvalidOperationExcpetion though.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101061
-----------------------------------------------------------
On Sept. 29, 2015, 6:59 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Sept. 29, 2015, 6:59 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Raju Bairishetti <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101061
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 1203)
<https://reviews.apache.org/r/38857/#comment158374>
Why are we changing exceptions here?
You may need to InvalidOperationExcpetion if you keep the older method declaration (i.e. throws MetaException, UnknownDBException, HiveSQLException, TException, LensException).
MetaException, UnknownDBException and InvalidOperationException all extends TException.
- Raju Bairishetti
On Sept. 29, 2015, 6:59 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Sept. 29, 2015, 6:59 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Deepak Barr <de...@gmail.com>.
> On Sept. 30, 2015, 6:24 a.m., Puneet Gupta wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1209
> > <https://reviews.apache.org/r/38857/diff/1/?file=1087309#file1087309line1209>
> >
> > Just a thought . Why cant we store MetaStoreClient as a private varaible in this method rather than getting it again and again by getting session first from a map and then the metastore client?
I agree.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101080
-----------------------------------------------------------
On Sept. 29, 2015, 6:59 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Sept. 29, 2015, 6:59 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Deepak Barr <de...@gmail.com>.
> On Sept. 30, 2015, 6:24 a.m., Puneet Gupta wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, lines 1208-1210
> > <https://reviews.apache.org/r/38857/diff/1/?file=1087309#file1087309line1208>
> >
> > Is it possible to use Filter instead to get required tables in one shot.
> > org.apache.hadoop.hive.metastore.IMetaStoreClient.listTableNamesByFilter(String, String, short).
> >
> > Not sure if we can create a specific filter for this case. To create such a filetr we should have a common key that exists for both native and non native tables and has a different value for each.
Hey Puneeet,
I considered that possibility but it seems the filter support is on limited keys. See https://hive.apache.org/javadocs/r0.12.0/api/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html#listTableNamesByFilter(java.lang.String, java.lang.String, short)
> On Sept. 30, 2015, 6:24 a.m., Puneet Gupta wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1201
> > <https://reviews.apache.org/r/38857/diff/1/?file=1087309#file1087309line1201>
> >
> > While we are at it, can we also change the name of this private method to indicate that it gets only NATIVE tables ?
> > Say: getNaviveTablesFromDB
I agree.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101080
-----------------------------------------------------------
On Sept. 29, 2015, 6:59 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Sept. 29, 2015, 6:59 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101080
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 1201)
<https://reviews.apache.org/r/38857/#comment158400>
While we are at it, can we also change the name of this private method to indicate that it gets only NATIVE tables ?
Say: getNaviveTablesFromDB
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 1203)
<https://reviews.apache.org/r/38857/#comment158401>
Again, while we are at it, can this method encapsulate all exceptions in LensException?
The cause execption,if any, can be attached to lens exception. This will be cleaner (unless we need special handling for any execption type , which is not the case as of now. NotFoundException has already been addressed earlier in the calling code) as the method will just throw LensException.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (lines 1208 - 1210)
<https://reviews.apache.org/r/38857/#comment158407>
Is it possible to use Filter instead to get required tables in one shot.
org.apache.hadoop.hive.metastore.IMetaStoreClient.listTableNamesByFilter(String, String, short).
Not sure if we can create a specific filter for this case. To create such a filetr we should have a common key that exists for both native and non native tables and has a different value for each.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 1209)
<https://reviews.apache.org/r/38857/#comment158402>
Just a thought . Why cant we store MetaStoreClient as a private varaible in this method rather than getting it again and again by getting session first from a map and then the metastore client?
- Puneet Gupta
On Sept. 29, 2015, 6:59 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Sept. 29, 2015, 6:59 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Deepak Barr <de...@gmail.com>.
> On Oct. 5, 2015, 8:50 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1228
> > <https://reviews.apache.org/r/38857/diff/3/?file=1088732#file1088732line1228>
> >
> > We might have to again split this up when we take up LENS-750. But I'm fine taking a decision as part of LENS-750
Thanks !
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101452
-----------------------------------------------------------
On Oct. 1, 2015, 6:52 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Oct. 1, 2015, 6:52 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/#review101452
-----------------------------------------------------------
Ship it!
Looks fine to me.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 1224)
<https://reviews.apache.org/r/38857/#comment158841>
We might have to again split this up when we take up LENS-750. But I'm fine taking a decision as part of LENS-750
- Amareshwari Sriramadasu
On Oct. 1, 2015, 6:52 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38857/
> -----------------------------------------------------------
>
> (Updated Oct. 1, 2015, 6:52 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> Currently, To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
>
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
>
>
> Diffs
> -----
>
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
>
> Diff: https://reviews.apache.org/r/38857/diff/
>
>
> Testing
> -------
>
> Yes.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
> [INFO] Lens ............................................... SUCCESS [ 4.601 s]
> [INFO] Lens API ........................................... SUCCESS [ 31.397 s]
> [INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
> [INFO] Lens Cube .......................................... SUCCESS [07:04 min]
> [INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
> [INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
> [INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
> [INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
> [INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
> [INFO] Lens Server ........................................ SUCCESS [11:11 min]
> [INFO] Lens client ........................................ SUCCESS [ 48.547 s]
> [INFO] Lens CLI ........................................... SUCCESS [03:45 min]
> [INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
> [INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
> [INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
> [INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
> [INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
> [INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 33:53 min
> [INFO] Finished at: 2015-09-29T22:48:22+05:30
> [INFO] Final Memory: 175M/3158M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Deepak Barr <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/
-----------------------------------------------------------
(Updated Oct. 1, 2015, 6:52 p.m.)
Review request for lens.
Changes
-------
Handled exceptions better.
Repository: lens
Description
-------
Currently, To obtain list of native tables, CubeMetastoreService does the following -
1. Fetches the list of tables ( one MetastoreClient call)
2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
Diffs (updated)
-----
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
Diff: https://reviews.apache.org/r/38857/diff/
Testing
-------
Yes.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
[INFO] Lens ............................................... SUCCESS [ 4.601 s]
[INFO] Lens API ........................................... SUCCESS [ 31.397 s]
[INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
[INFO] Lens Cube .......................................... SUCCESS [07:04 min]
[INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
[INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
[INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
[INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
[INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
[INFO] Lens Server ........................................ SUCCESS [11:11 min]
[INFO] Lens client ........................................ SUCCESS [ 48.547 s]
[INFO] Lens CLI ........................................... SUCCESS [03:45 min]
[INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
[INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
[INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
[INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
[INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
[INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:53 min
[INFO] Finished at: 2015-09-29T22:48:22+05:30
[INFO] Final Memory: 175M/3158M
[INFO] ------------------------------------------------------------------------
Thanks,
Deepak Barr
Re: Review Request 38857: LENS-796 : Slow response times for
/metastore/nativetables API
Posted by Deepak Barr <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38857/
-----------------------------------------------------------
(Updated Sept. 30, 2015, 11:30 a.m.)
Review request for lens.
Changes
-------
Updated diff -
1. Reused Metastore client
2. Changed method name
3. Reverted back the exceptions thrown and handled invalidoperationexception.
Repository: lens
Description
-------
Currently, To obtain list of native tables, CubeMetastoreService does the following -
1. Fetches the list of tables ( one MetastoreClient call)
2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
Diffs (updated)
-----
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 37cebfe
Diff: https://reviews.apache.org/r/38857/diff/
Testing
-------
Yes.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.745 s]
[INFO] Lens ............................................... SUCCESS [ 4.601 s]
[INFO] Lens API ........................................... SUCCESS [ 31.397 s]
[INFO] Lens API for server and extensions ................. SUCCESS [ 28.560 s]
[INFO] Lens Cube .......................................... SUCCESS [07:04 min]
[INFO] Lens DB storage .................................... SUCCESS [ 27.465 s]
[INFO] Lens Query Library ................................. SUCCESS [ 23.361 s]
[INFO] Lens Hive Driver ................................... SUCCESS [04:14 min]
[INFO] Lens Driver for JDBC ............................... SUCCESS [ 50.792 s]
[INFO] Lens Elastic Search Driver ......................... SUCCESS [ 20.551 s]
[INFO] Lens Server ........................................ SUCCESS [11:11 min]
[INFO] Lens client ........................................ SUCCESS [ 48.547 s]
[INFO] Lens CLI ........................................... SUCCESS [03:45 min]
[INFO] Lens Examples ...................................... SUCCESS [ 11.595 s]
[INFO] Lens Distribution .................................. SUCCESS [ 13.176 s]
[INFO] Lens ML Lib ........................................ SUCCESS [02:03 min]
[INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.752 s]
[INFO] Lens Regression .................................... SUCCESS [ 15.811 s]
[INFO] Lens UI ............................................ SUCCESS [ 50.811 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:53 min
[INFO] Finished at: 2015-09-29T22:48:22+05:30
[INFO] Final Memory: 175M/3158M
[INFO] ------------------------------------------------------------------------
Thanks,
Deepak Barr