You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Sushil Mohanty <su...@gmail.com> on 2016/10/14 08:29:24 UTC
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/
-----------------------------------------------------------
(Updated Oct. 14, 2016, 8:29 a.m.)
Review request for lens.
Changes
-------
- Added support for HDFS as DB resource path.
- Addressed other review comments.
Bugs: LENS-317
https://issues.apache.org/jira/browse/LENS-317
Repository: lens
Description
-------
Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
Diffs (updated)
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8cf617b
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
lens-server/pom.xml 6dea9a7
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
lens-server/src/test/java/org/apache/lens/server/metastore/TestDatabaseService.java PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
lens-server/src/test/resources/lens-site.xml d96659f
Diff: https://reviews.apache.org/r/51964/diff/
Testing
-------
mvn clean install.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
[INFO] Lens .............................................. SUCCESS [7.286s]
[INFO] Lens API .......................................... SUCCESS [31.546s]
[INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
[INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
[INFO] Lens DB storage ................................... SUCCESS [25.650s]
[INFO] Lens Query Library ................................ SUCCESS [21.646s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
[INFO] Lens Server ....................................... SUCCESS [18:19.588s]
[INFO] Lens client ....................................... SUCCESS [2:02.590s]
[INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
[INFO] Lens Examples ..................................... SUCCESS [13.790s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
[INFO] Lens Distribution ................................. SUCCESS [24.869s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
[INFO] Lens Regression ................................... SUCCESS [20.677s]
[INFO] Lens UI ........................................... SUCCESS [19.454s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:44.002s
[INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
[INFO] Final Memory: 178M/2490M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
> On Oct. 18, 2016, 2:09 a.m., Puneet Gupta wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 1397
> > <https://reviews.apache.org/r/51964/diff/2/?file=1535808#file1535808line1397>
> >
> > Are two commands required for Hive ?
yes, for health check test case.
> On Oct. 18, 2016, 2:09 a.m., Puneet Gupta wrote:
> > lens-client/src/main/java/org/apache/lens/client/LensConnection.java, lines 395-398
> > <https://reviews.apache.org/r/51964/diff/2/?file=1535804#file1535804line395>
> >
> > How do we handle non jar type resources both on client and on server ?
We are supporting only jar files as of now. In server side we have check for file type.
> On Oct. 18, 2016, 2:09 a.m., Puneet Gupta wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, lines 762-764
> > <https://reviews.apache.org/r/51964/diff/2/?file=1535808#file1535808line762>
> >
> > How is addJar updating org.apache.lens.server.session.DatabaseResourceService#classLoaderCache with calssloader pointing to the newly uploaded jar? Will the new jar be refletced in exsitin session calss loader as well ?
Added method addLocalDbResourceEntry() which will be taken care in loadMappedResources().
- Sushil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review152818
-----------------------------------------------------------
On Oct. 14, 2016, 8:29 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Oct. 14, 2016, 8:29 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8cf617b
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server/pom.xml 6dea9a7
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/metastore/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/resources/lens-site.xml d96659f
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review152818
-----------------------------------------------------------
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java (line 134)
<https://reviews.apache.org/r/51964/#comment222229>
How are we handling HDFS path ? are we downloading this jar to local machine (hosting the cli) and then uploading to server ?
lens-client/src/main/java/org/apache/lens/client/LensConnection.java (lines 393 - 396)
<https://reviews.apache.org/r/51964/#comment222230>
How do we handle non jar type resources both on client and on server ?
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 738)
<https://reviews.apache.org/r/51964/#comment222046>
Should we have logs statements at the start and end of this method as its syncronized so that we know why the other addjar call is waiting (if that situation ever arises)
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 760)
<https://reviews.apache.org/r/51964/#comment222225>
Do we need to update org.apache.lens.server.session.DatabaseResourceService#dbResEntryMap with entry in this case?
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (lines 762 - 764)
<https://reviews.apache.org/r/51964/#comment222228>
How is addJar updating org.apache.lens.server.session.DatabaseResourceService#classLoaderCache with calssloader pointing to the newly uploaded jar? Will the new jar be refletced in exsitin session calss loader as well ?
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (lines 792 - 795)
<https://reviews.apache.org/r/51964/#comment222042>
Should we fail in this case or create the directory and add the jar? I feel, the use case where a user creates a new DB from lens CLI and then uploads a jar should work without admin intervention.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 801)
<https://reviews.apache.org/r/51964/#comment222044>
Should we show a different message to user which says
"This database {dbname} does not support jar upload"
The warning message can still have details that jar_oder file is present and hence upload is not allowed
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (lines 808 - 812)
<https://reviews.apache.org/r/51964/#comment222048>
Theoratically this case should not arive since the method is syncronized.
If this does occur, it can be because server was resttarted while a jar was still uploading or rename opertaion failed. In this case(s) we should overwrite the jar with a warning message.
Please check once.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (lines 815 - 827)
<https://reviews.apache.org/r/51964/#comment222204>
should we move this logic to a utility and use it for both DatabaseResourceService and CubeMetastoreServiceImpl ?
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 829)
<https://reviews.apache.org/r/51964/#comment222205>
This seems same as "uploadingPath"
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 834)
<https://reviews.apache.org/r/51964/#comment222211>
Can we resuse "dbDir" in all paces where DB path is required
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (lines 840 - 845)
<https://reviews.apache.org/r/51964/#comment222208>
Should we club the two exceptions (catch with multiple exceptions) and just say "Execption while uploading jar". The stack trace will have the exception type and details anyway.
Or we can even have smaller try blocks and print the exact execption message (Say a try block around IOUtils.copy(is, fos) and the catch says execption wile copying jar)
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 1397)
<https://reviews.apache.org/r/51964/#comment222216>
Are two commands required for Hive ?
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java (lines 1566 - 1567)
<https://reviews.apache.org/r/51964/#comment222217>
fileDetail needs to me added too ?
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java (line 1576)
<https://reviews.apache.org/r/51964/#comment222219>
can only JARs be added as resources ? If yes then should we rename the method to include jar
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java (line 1581)
<https://reviews.apache.org/r/51964/#comment222232>
This file size is passed by the client and can be dummy size. Do we need to reconfirm theactual size on server (This can be done while copying jar to local)?
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java (lines 1582 - 1584)
<https://reviews.apache.org/r/51964/#comment222231>
Space required before FileSize and AllowedSize
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 51)
<https://reviews.apache.org/r/51964/#comment221940>
can we update the variable name to match the ds
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 84)
<https://reviews.apache.org/r/51964/#comment221943>
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 91)
<https://reviews.apache.org/r/51964/#comment221947>
should we call this in the end ? I feel, the INITED state can be flagged after local initialization.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 113 - 116)
<https://reviews.apache.org/r/51964/#comment221950>
else part can be skipped. resTopDir is already initialized. If it makes sense , we can move this initialization (resTopDir = getHiveConf().get(LensConfConstants.DATABASE_RESOURCE_DIR...)) to init() method also.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 121)
<https://reviews.apache.org/r/51964/#comment221977>
Should we change this to "DB resources base location does not exist" ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 125)
<https://reviews.apache.org/r/51964/#comment221978>
Should we chnage the message to "Error locating DB resources base directory" ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 192)
<https://reviews.apache.org/r/51964/#comment222001>
can we log "dbResEntryMap.get(dbname)" instaed
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 295)
<https://reviews.apache.org/r/51964/#comment222037>
can we also add DB name in error message
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 415)
<https://reviews.apache.org/r/51964/#comment222223>
Should we remove Map from method name
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java (line 429)
<https://reviews.apache.org/r/51964/#comment222222>
When is readClassLoaderFromCache false ?
- Puneet Gupta
On Oct. 14, 2016, 8:29 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Oct. 14, 2016, 8:29 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8cf617b
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server/pom.xml 6dea9a7
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/metastore/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/resources/lens-site.xml d96659f
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
> On Oct. 26, 2016, 6:01 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java, line 339
> > <https://reviews.apache.org/r/51964/diff/4/?file=1545071#file1545071line339>
> >
> > Why are we passing null name? We should remove the param, if it is not required anymore.
Moved NAME to class level variable and CLIService is passed as null. Super class BaseLensService expects CLIService as param.
> On Oct. 26, 2016, 6:01 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java, line 169
> > <https://reviews.apache.org/r/51964/diff/4/?file=1545070#file1545070line169>
> >
> > not sure how its corresponding local jar file is added to classloader. If hdfs jar is getting added, it wont work.
> >
> > might have to pass the corresponding local jar as the location value for ResourceEntry.
In case of base dir is hdfs, first copying the files to local directory and setting local as base directory and respective jar files are picked from here.
if (resTopDirPath.getFileSystem(getHiveConf()).getScheme().equals("hdfs")) {
localResTopDir = getHiveConf().get(LensConfConstants.DATABASE_LOCAL_RESOURCE_DIR,
LensConfConstants.DEFAULT_LOCAL_DATABASE_RESOURCE_DIR);
downloadJarFilesFromHDFS(resTopDir, localResTopDir);
resTopDir = localResTopDir;
}
- Sushil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review153851
-----------------------------------------------------------
On Oct. 25, 2016, 10:11 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Oct. 25, 2016, 10:11 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 3ae59c6
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml 8e5fe8a
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 4928c1f
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
> On Oct. 26, 2016, 6:01 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java, line 169
> > <https://reviews.apache.org/r/51964/diff/4/?file=1545070#file1545070line169>
> >
> > not sure how its corresponding local jar file is added to classloader. If hdfs jar is getting added, it wont work.
> >
> > might have to pass the corresponding local jar as the location value for ResourceEntry.
>
> Sushil Mohanty wrote:
> In case of base dir is hdfs, first copying the files to local directory and setting local as base directory and respective jar files are picked from here.
>
> if (resTopDirPath.getFileSystem(getHiveConf()).getScheme().equals("hdfs")) {
> localResTopDir = getHiveConf().get(LensConfConstants.DATABASE_LOCAL_RESOURCE_DIR,
> LensConfConstants.DEFAULT_LOCAL_DATABASE_RESOURCE_DIR);
> downloadJarFilesFromHDFS(resTopDir, localResTopDir);
> resTopDir = localResTopDir;
> }
>
> Amareshwari Sriramadasu wrote:
> We cannot change resTopDir which is hdfs to a local path.
Addressed.
- Sushil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review153851
-----------------------------------------------------------
On Nov. 3, 2016, 5:31 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 5:31 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
> lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml a00048b
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 29c59d3
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Oct. 26, 2016, 6:01 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java, line 169
> > <https://reviews.apache.org/r/51964/diff/4/?file=1545070#file1545070line169>
> >
> > not sure how its corresponding local jar file is added to classloader. If hdfs jar is getting added, it wont work.
> >
> > might have to pass the corresponding local jar as the location value for ResourceEntry.
>
> Sushil Mohanty wrote:
> In case of base dir is hdfs, first copying the files to local directory and setting local as base directory and respective jar files are picked from here.
>
> if (resTopDirPath.getFileSystem(getHiveConf()).getScheme().equals("hdfs")) {
> localResTopDir = getHiveConf().get(LensConfConstants.DATABASE_LOCAL_RESOURCE_DIR,
> LensConfConstants.DEFAULT_LOCAL_DATABASE_RESOURCE_DIR);
> downloadJarFilesFromHDFS(resTopDir, localResTopDir);
> resTopDir = localResTopDir;
> }
We cannot change resTopDir which is hdfs to a local path.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review153851
-----------------------------------------------------------
On Nov. 3, 2016, 5:31 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 5:31 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
> lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml a00048b
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 29c59d3
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review153851
-----------------------------------------------------------
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java (line 135)
<https://reviews.apache.org/r/51964/#comment223366>
Make the argument File instead of String.
lens-client/src/main/java/org/apache/lens/client/LensConnection.java (line 372)
<https://reviews.apache.org/r/51964/#comment223367>
Please add unit test for the client api/cli command
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java (line 841)
<https://reviews.apache.org/r/51964/#comment223368>
local directory need not be qualified, as it will be always local.
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java (line 23)
<https://reviews.apache.org/r/51964/#comment223369>
required import?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 90)
<https://reviews.apache.org/r/51964/#comment223370>
Can we store resourceFs which is a FileSystem object as class variable, instead of creating newInstance on every download ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 109)
<https://reviews.apache.org/r/51964/#comment223371>
Lets move super.start() towards end of this method.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 117 - 130)
<https://reviews.apache.org/r/51964/#comment223372>
Does server start has to download/refresh jars upon every startup ? It would be expensive. and we are doing even if there are no changes to the files.
Should we do size checks across local and remote filesystem before any copying?
Put this enabled/disabled with a configuration?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 123 - 129)
<https://reviews.apache.org/r/51964/#comment223373>
This check has to happen before the above copy.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 169)
<https://reviews.apache.org/r/51964/#comment223374>
not sure how its corresponding local jar file is added to classloader. If hdfs jar is getting added, it wont work.
might have to pass the corresponding local jar as the location value for ResourceEntry.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 213)
<https://reviews.apache.org/r/51964/#comment223375>
Can we store this at class level variable instead of reading from conf again and again ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 547 - 567)
<https://reviews.apache.org/r/51964/#comment223376>
We should not be filesystem close anywhere, FileSystem's shutdown hook will take care of it.
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java (line 339)
<https://reviews.apache.org/r/51964/#comment223377>
Why are we passing null name? We should remove the param, if it is not required anymore.
lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java (line 310)
<https://reviews.apache.org/r/51964/#comment223378>
what is partial state here? I dont see any return with partial state.
lens-server/src/main/resources/lensserver-default.xml (line 58)
<https://reviews.apache.org/r/51964/#comment223379>
Do we need this change? I see DatabaseResourceService is still getting added through SessionService.
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java (line 174)
<https://reviews.apache.org/r/51964/#comment223380>
Can you change logging style to slf4j parameterized style?
- Amareshwari Sriramadasu
On Oct. 25, 2016, 10:11 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Oct. 25, 2016, 10:11 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 3ae59c6
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml 8e5fe8a
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 4928c1f
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
> On Nov. 4, 2016, 6:35 a.m., Amareshwari Sriramadasu wrote:
> > lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java, line 97
> > <https://reviews.apache.org/r/51964/diff/5/?file=1553355#file1553355line97>
> >
> > Database does not exists, but jar can be added?
It was being added to the default db. Changed it to work with specified db.
- Sushil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review154869
-----------------------------------------------------------
On Nov. 3, 2016, 5:31 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 5:31 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
> lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml a00048b
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 29c59d3
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Nov. 4, 2016, 6:35 a.m., Amareshwari Sriramadasu wrote:
> > lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java, line 104
> > <https://reviews.apache.org/r/51964/diff/5/?file=1553355#file1553355line104>
> >
> > command should be throwing NullPointerException on non existing jar. It should be a proper exception with message "File does not exist".
I meant 'Command should not be throwing NullPointerException '
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review154869
-----------------------------------------------------------
On Nov. 3, 2016, 5:31 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 5:31 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
> lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml a00048b
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 29c59d3
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review154869
-----------------------------------------------------------
lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java (line 97)
<https://reviews.apache.org/r/51964/#comment224650>
Database does not exists, but jar can be added?
lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java (line 99)
<https://reviews.apache.org/r/51964/#comment224648>
We cannot checkin a jar file into the source code. Please generate jar file at compile time. You can see lens-server/pom.xml on how test jars are generated.
lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java (line 104)
<https://reviews.apache.org/r/51964/#comment224652>
command should be throwing NullPointerException on non existing jar. It should be a proper exception with message "File does not exist".
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 149)
<https://reviews.apache.org/r/51964/#comment224655>
Move serverFs also to class level? And shall we call it dbResourceFs instead ?
lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java (line 82)
<https://reviews.apache.org/r/51964/#comment224658>
We should access the instance in sessionService, instead of LensServices. Inthat case you wont even need it as class variable.
lens-server/src/test/resources/lens-site.xml (line 149)
<https://reviews.apache.org/r/51964/#comment224657>
We should not be doing this. As DatabaseResourceService is getting created in SessionService already, this will make two service instances to be created.
- Amareshwari Sriramadasu
On Nov. 3, 2016, 5:31 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2016, 5:31 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
> lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml a00048b
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 29c59d3
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review156012
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 52)
<https://reviews.apache.org/r/51964/#comment226164>
classLoaderCache needs to be updated on addDBJar.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 114)
<https://reviews.apache.org/r/51964/#comment226142>
This method should be private .
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 265 - 268)
<https://reviews.apache.org/r/51964/#comment226143>
This method should be private
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 408 - 414)
<https://reviews.apache.org/r/51964/#comment226136>
Should we decide which map to use based on mode (local/hdfs or remote ) which is decided on startup
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 416)
<https://reviews.apache.org/r/51964/#comment226145>
addremoteDbResourceEntry and addLocalDbResourceEntry can be private .
We can also decide and remove these methods and directly add map entries in calling code.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 420)
<https://reviews.apache.org/r/51964/#comment226111>
Should we call this method updateLocalDBResourceEntries since its replacing all the entries for the DB ?
Simlaraly updateRemoteDbResourceEntries too
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 434)
<https://reviews.apache.org/r/51964/#comment226165>
QueryExecutionServiceImpl#addSessionResourcesToDriver needs to remove old jars added to the hive sessions and add the newly uploaded jar to it (there is one hive session per User and per DB. Only the current DB sessions should get affected)
This implies that there is no guarantee on success of currently running queries on the affected Hive sessions
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 436)
<https://reviews.apache.org/r/51964/#comment226095>
Should we also add DB name to this log ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 476)
<https://reviews.apache.org/r/51964/#comment226108>
jarFilePath has the complete path for the jar file. Do we need currentDB and baseDir ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 484)
<https://reviews.apache.org/r/51964/#comment226132>
should we check the status of this opeartion and throw exception in case of failure.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 505)
<https://reviews.apache.org/r/51964/#comment226096>
should we resuse FileSystem created while starting the service ? This can be done for all FieSystem.newInstance calls in this method
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 508)
<https://reviews.apache.org/r/51964/#comment226098>
should we also check the boolean value returned by this call and proceed accrodingly
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 522)
<https://reviews.apache.org/r/51964/#comment226101>
Should we use FileSystem.delete(path,false) instead and check the boolean value returned by this call before proceeding further?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 535)
<https://reviews.apache.org/r/51964/#comment226135>
Should we INFO log that jar uploaded to {path}
- Puneet Gupta
On Nov. 8, 2016, 10:18 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2016, 10:18 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/pom.xml 8e5e3eb
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
> lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml a00048b
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/session/TestSessionClassLoaders.java d66de4c
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 29c59d3
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review155416
-----------------------------------------------------------
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java (lines 837 - 849)
<https://reviews.apache.org/r/51964/#comment225885>
Should we add the new properties to default lens server config file along with the description on how and when to use them?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 84)
<https://reviews.apache.org/r/51964/#comment225881>
[Suggestion] Should we simplify this method by assuming there will be only one level of folders under the base resource directory ?
If we use this knowledge we can sync one DB at a time and paring the urls may not be required.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 87)
<https://reviews.apache.org/r/51964/#comment225769>
Can we reuse "dbResourceFs" ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 97 - 109)
<https://reviews.apache.org/r/51964/#comment225886>
Can we add log entries when jars are downloaded
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 143)
<https://reviews.apache.org/r/51964/#comment225349>
This needs to be moved to end so that service state is changed to STARTED in the end .
org.apache.hive.service.AbstractService#start
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 157 - 159)
<https://reviews.apache.org/r/51964/#comment225350>
What will happen in case the scheme is not hdfs?
Will the flow work gracefully or throw an execption
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 161 - 163)
<https://reviews.apache.org/r/51964/#comment225890>
What do we do for use cases where the user does not need any DB resources ? In those cases it may not be an ERROR to not have top resource directory
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 193 - 194)
<https://reviews.apache.org/r/51964/#comment225888>
Can we set dbLocalResTopDir to dbResTopDir when dbResTopDir is local path. This check would not be required then.
OR
If needed we can also store the mode in start method as a boolean (local/hdfs) locally and use it in mapCommonResourceEntries(boolean mode) and mapDbResourceEntries(boolean mode)
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 210 - 211)
<https://reviews.apache.org/r/51964/#comment225893>
This will never be true since the DB path is got by listing the base directory.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 231)
<https://reviews.apache.org/r/51964/#comment225889>
Can we create the loacl fileSystem object only once (may be in start() method and) use it elsewhere as hadoop FileSystem object is a heavy and mainatins state/cache inside it.
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (lines 233 - 237)
<https://reviews.apache.org/r/51964/#comment225892>
Isn't this condition already checked in the calling method ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 265)
<https://reviews.apache.org/r/51964/#comment225984>
Can we just retun the jar path/name or null in case there is no jar in the folder and shouldIncrVersion = false
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 272)
<https://reviews.apache.org/r/51964/#comment225980>
Can we reuase FileSystem instance here too ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 278)
<https://reviews.apache.org/r/51964/#comment225981>
Should this be >=1 ?
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java (line 446)
<https://reviews.apache.org/r/51964/#comment225985>
Should we use LensConfConstants.DB_RESOURCE_COPY_FROM_HDFS property to figure out HDFS vs Local ?
We can rename the property to DB_RESOURCES_IN_HDFS = true/false?
This review only covers the startup flow. addDBJar Flow Review is pending, will review it soon.
- Puneet Gupta
On Nov. 8, 2016, 10:18 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2016, 10:18 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/pom.xml 8e5e3eb
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
> lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
> lens-server/pom.xml d24dc1e
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/main/resources/lensserver-default.xml a00048b
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/session/TestSessionClassLoaders.java d66de4c
> lens-server/src/test/resources/lens-site.xml d96659f
> pom.xml 29c59d3
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/
-----------------------------------------------------------
(Updated Nov. 8, 2016, 10:18 a.m.)
Review request for lens.
Changes
-------
- Addressed review.
Bugs: LENS-317
https://issues.apache.org/jira/browse/LENS-317
Repository: lens
Description
-------
Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
Diffs (updated)
-----
lens-cli/pom.xml 8e5e3eb
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
lens-server/pom.xml d24dc1e
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
lens-server/src/main/resources/lensserver-default.xml a00048b
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/session/TestSessionClassLoaders.java d66de4c
lens-server/src/test/resources/lens-site.xml d96659f
pom.xml 29c59d3
Diff: https://reviews.apache.org/r/51964/diff/
Testing
-------
mvn clean install.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
[INFO] Lens .............................................. SUCCESS [7.286s]
[INFO] Lens API .......................................... SUCCESS [31.546s]
[INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
[INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
[INFO] Lens DB storage ................................... SUCCESS [25.650s]
[INFO] Lens Query Library ................................ SUCCESS [21.646s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
[INFO] Lens Server ....................................... SUCCESS [18:19.588s]
[INFO] Lens client ....................................... SUCCESS [2:02.590s]
[INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
[INFO] Lens Examples ..................................... SUCCESS [13.790s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
[INFO] Lens Distribution ................................. SUCCESS [24.869s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
[INFO] Lens Regression ................................... SUCCESS [20.677s]
[INFO] Lens UI ........................................... SUCCESS [19.454s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:44.002s
[INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
[INFO] Final Memory: 178M/2490M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/
-----------------------------------------------------------
(Updated Nov. 3, 2016, 5:31 p.m.)
Review request for lens.
Changes
-------
- Addressed review.
Bugs: LENS-317
https://issues.apache.org/jira/browse/LENS-317
Repository: lens
Description
-------
Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
Diffs (updated)
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
lens-cli/src/test/java/org/apache/lens/cli/TestLensDatabaseCommands.java 7fc8438
lens-cli/src/test/resources/schema/jars/test_db_resource.jar PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8f9db2a
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
lens-server/pom.xml d24dc1e
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
lens-server/src/main/resources/lensserver-default.xml a00048b
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
lens-server/src/test/resources/lens-site.xml d96659f
pom.xml 29c59d3
Diff: https://reviews.apache.org/r/51964/diff/
Testing
-------
mvn clean install.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
[INFO] Lens .............................................. SUCCESS [7.286s]
[INFO] Lens API .......................................... SUCCESS [31.546s]
[INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
[INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
[INFO] Lens DB storage ................................... SUCCESS [25.650s]
[INFO] Lens Query Library ................................ SUCCESS [21.646s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
[INFO] Lens Server ....................................... SUCCESS [18:19.588s]
[INFO] Lens client ....................................... SUCCESS [2:02.590s]
[INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
[INFO] Lens Examples ..................................... SUCCESS [13.790s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
[INFO] Lens Distribution ................................. SUCCESS [24.869s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
[INFO] Lens Regression ................................... SUCCESS [20.677s]
[INFO] Lens UI ........................................... SUCCESS [19.454s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:44.002s
[INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
[INFO] Final Memory: 178M/2490M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/
-----------------------------------------------------------
(Updated Oct. 25, 2016, 10:11 a.m.)
Review request for lens.
Changes
-------
Addressed review comments.
Bugs: LENS-317
https://issues.apache.org/jira/browse/LENS-317
Repository: lens
Description
-------
Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
Diffs (updated)
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 3ae59c6
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java 20ec686
lens-server/pom.xml d24dc1e
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java 21e2a62
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
lens-server/src/main/java/org/apache/lens/server/session/SessionResource.java 63eea63
lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
lens-server/src/main/resources/lensserver-default.xml 8e5fe8a
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseService.java PRE-CREATION
lens-server/src/test/resources/lens-site.xml d96659f
pom.xml 4928c1f
Diff: https://reviews.apache.org/r/51964/diff/
Testing
-------
mvn clean install.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
[INFO] Lens .............................................. SUCCESS [7.286s]
[INFO] Lens API .......................................... SUCCESS [31.546s]
[INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
[INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
[INFO] Lens DB storage ................................... SUCCESS [25.650s]
[INFO] Lens Query Library ................................ SUCCESS [21.646s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
[INFO] Lens Server ....................................... SUCCESS [18:19.588s]
[INFO] Lens client ....................................... SUCCESS [2:02.590s]
[INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
[INFO] Lens Examples ..................................... SUCCESS [13.790s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
[INFO] Lens Distribution ................................. SUCCESS [24.869s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
[INFO] Lens Regression ................................... SUCCESS [20.677s]
[INFO] Lens UI ........................................... SUCCESS [19.454s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:44.002s
[INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
[INFO] Final Memory: 178M/2490M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
> On Oct. 21, 2016, 1:03 p.m., Amareshwari Sriramadasu wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java, line 576
> > <https://reviews.apache.org/r/51964/diff/3/?file=1542202#file1542202line576>
> >
> > Just a thought - Should this be part of SessionService?
Yes moved to SessionService from MetastoreService
> On Oct. 21, 2016, 1:03 p.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java, line 751
> > <https://reviews.apache.org/r/51964/diff/3/?file=1542204#file1542204line751>
> >
> > We shouldnt be checking for scheme starting with hdfs:// as a string. We should convert the directory as path and check if it is localfile system or not.
Didn't find any method to check whether FileSystem is local or not, instead used FileSystem.getScheme().
- Sushil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review153535
-----------------------------------------------------------
On Oct. 20, 2016, 1:24 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2016, 1:24 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8cf617b
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server/pom.xml 6dea9a7
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/metastore/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/resources/lens-site.xml d96659f
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/#review153535
-----------------------------------------------------------
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java (line 837)
<https://reviews.apache.org/r/51964/#comment222929>
Should we qualify the default path now with file:/// ?
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java (line 576)
<https://reviews.apache.org/r/51964/#comment222930>
Just a thought - Should this be part of SessionService?
lens-server/pom.xml (line 275)
<https://reviews.apache.org/r/51964/#comment222928>
Can you add the dependency to main pom's dependency management section? And version, scope and exclusions should go there.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 741)
<https://reviews.apache.org/r/51964/#comment222931>
We should remove calendar time from logging, as logger would already put time.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (lines 745 - 764)
<https://reviews.apache.org/r/51964/#comment222934>
We should move all this code to DBResourceService.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 751)
<https://reviews.apache.org/r/51964/#comment222933>
We shouldnt be checking for scheme starting with hdfs:// as a string. We should convert the directory as path and check if it is localfile system or not.
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java (line 765)
<https://reviews.apache.org/r/51964/#comment222932>
We should remove calendar time from logging, as logger would already put time.
- Amareshwari Sriramadasu
On Oct. 20, 2016, 1:24 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51964/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2016, 1:24 p.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-317
> https://issues.apache.org/jira/browse/LENS-317
>
>
> Repository: lens
>
>
> Description
> -------
>
> Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
>
>
> Diffs
> -----
>
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
> lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
> lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8cf617b
> lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
> lens-server/pom.xml 6dea9a7
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
> lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
> lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
> lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
> lens-server/src/test/java/org/apache/lens/server/metastore/TestDatabaseService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
> lens-server/src/test/resources/lens-site.xml d96659f
>
> Diff: https://reviews.apache.org/r/51964/diff/
>
>
> Testing
> -------
>
> mvn clean install.
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
> [INFO] Lens .............................................. SUCCESS [7.286s]
> [INFO] Lens API .......................................... SUCCESS [31.546s]
> [INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
> [INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
> [INFO] Lens DB storage ................................... SUCCESS [25.650s]
> [INFO] Lens Query Library ................................ SUCCESS [21.646s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
> [INFO] Lens Server ....................................... SUCCESS [18:19.588s]
> [INFO] Lens client ....................................... SUCCESS [2:02.590s]
> [INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
> [INFO] Lens Examples ..................................... SUCCESS [13.790s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
> [INFO] Lens Distribution ................................. SUCCESS [24.869s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
> [INFO] Lens Regression ................................... SUCCESS [20.677s]
> [INFO] Lens UI ........................................... SUCCESS [19.454s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 49:44.002s
> [INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
> [INFO] Final Memory: 178M/2490M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 51964: LENS-317: Server side calls for updating
database jars
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51964/
-----------------------------------------------------------
(Updated Oct. 20, 2016, 1:24 p.m.)
Review request for lens.
Changes
-------
Addressed review comments.
Bugs: LENS-317
https://issues.apache.org/jira/browse/LENS-317
Repository: lens
Description
-------
Server side api call to update database jar without restarting lens server. More details can be found in LENS-317.
Diffs (updated)
-----
lens-cli/src/main/java/org/apache/lens/cli/commands/LensDatabaseCommands.java c6ae02b
lens-client/src/main/java/org/apache/lens/client/LensClient.java e936798
lens-client/src/main/java/org/apache/lens/client/LensConnection.java bb15b23
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 8cf617b
lens-server-api/src/main/java/org/apache/lens/server/api/metastore/CubeMetastoreService.java 28b9d22
lens-server/pom.xml 6dea9a7
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 8b10d1d
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java 9d823da
lens-server/src/main/java/org/apache/lens/server/session/DatabaseResourceService.java 511e4cf
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 34c901c
lens-server/src/main/java/org/apache/lens/server/util/ScannedPaths.java e48eab4
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 7cccf30
lens-server/src/test/java/org/apache/lens/server/LensServerTestUtil.java 67cee57
lens-server/src/test/java/org/apache/lens/server/metastore/TestDatabaseService.java PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/session/TestDatabaseResourceService.java 2bc3712
lens-server/src/test/resources/lens-site.xml d96659f
Diff: https://reviews.apache.org/r/51964/diff/
Testing
-------
mvn clean install.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [3.302s]
[INFO] Lens .............................................. SUCCESS [7.286s]
[INFO] Lens API .......................................... SUCCESS [31.546s]
[INFO] Lens API for server and extensions ................ SUCCESS [25.681s]
[INFO] Lens Cube ......................................... SUCCESS [17:55.255s]
[INFO] Lens DB storage ................................... SUCCESS [25.650s]
[INFO] Lens Query Library ................................ SUCCESS [21.646s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:11.167s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:03.464s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [54.798s]
[INFO] Lens Server ....................................... SUCCESS [18:19.588s]
[INFO] Lens client ....................................... SUCCESS [2:02.590s]
[INFO] Lens CLI .......................................... SUCCESS [1:54.985s]
[INFO] Lens Examples ..................................... SUCCESS [13.790s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [2.061s]
[INFO] Lens Distribution ................................. SUCCESS [24.869s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:54.426s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.053s]
[INFO] Lens Regression ................................... SUCCESS [20.677s]
[INFO] Lens UI ........................................... SUCCESS [19.454s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:44.002s
[INFO] Finished at: Sat Sep 17 00:12:08 IST 2016
[INFO] Final Memory: 178M/2490M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty