[jira] [Commented] (LENS-408) Creating HiveConf multiple times in estimate flow should be avoided

["Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LENS-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362749#comment-14362749 ] 

Amareshwari Sriramadasu commented on LENS-408:
----------------------------------------------

Review request : https://reviews.apache.org/r/32043/

After the changes put up, following are all the occurrences of HiveConf creation in code.

* No occurrences
{noformat}
lens-api amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
lens-cli amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
lens-client amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
lens-driver-jdbc amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
ens-storage-db amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
lens-query-lib amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
lens-examples amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
{noformat}
* lens-cube module :
{noformat}
lens-cube amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
src/main/java/org/apache/lens/cube/metadata/CubeMetastoreClient.java:    this.config = new HiveConf(conf);
src/main/java/org/apache/lens/cube/parse/HQLParser.java:    ASTNode ast = parseHQL("select * from default_table ", new HiveConf());
{noformat}
** There is HiveConf created when creating CubeMetastoreClient. One call per db
** The one HQLParser.java is in main() method. mainly for debugging purpose. Does not come usual code flow.
* Hive Driver module :
{noformat}
lens-driver-hive amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
src/main/java/org/apache/lens/driver/hive/HiveDriver.java:    // resources have to be added separately on hiveConf again because new HiveConf() overrides hive.* properties
src/main/java/org/apache/lens/driver/hive/HiveDriver.java:    this.hiveConf = new HiveConf(driverConf, HiveDriver.class);
src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java:    // new HiveConf() is getting created because connection will be different for each user
src/main/java/org/apache/lens/driver/hive/RemoteThriftConnection.java:    this.conf = new HiveConf(conf);
{noformat}
** One occurrence is during driver initialization
** One occurrence is during open connection with HiveServer - will be called once per lens session and thread.
* Created LENS-409 for tracking occurrences on lens-ml-lib
* lens-server module :
{noformat}
lens-server amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
src/main/java/org/apache/lens/server/LensServerConf.java:    public static final HiveConf CONF = new HiveConf();
src/main/java/org/apache/lens/server/LensServerConf.java:    return new HiveConf(ConfHolder.CONF);
{noformat}
** Once during server initialization
** One will be called from base class of LensSessionImpl - once per session.
** The one in LensServerConf  method is used in tests
* lens-server-api method :
{noformat}
lens-server-api amareshwari.sr$ git grep 'new HiveConf(' src/main/java/
src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java:      hiveConf = new HiveConf(this.conf, this.getClass());
{noformat}
** Once per query.


> Creating HiveConf multiple times in estimate flow should be avoided
> -------------------------------------------------------------------
>
>                 Key: LENS-408
>                 URL: https://issues.apache.org/jira/browse/LENS-408
>             Project: Apache Lens
>          Issue Type: Improvement
>          Components: driver-hive, driver-jdbc, server
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 2.1
>
>
> We are seeing creation of new HiveConf() is taking around 20 milli seconds on the average.  And is called in many places in code without much need. 
> This issue is to look at all such occurrences and fix them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)