You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Puneet Gupta <pu...@gmail.com> on 2015/11/02 05:08:09 UTC
Review Request 39842: Ability to load different instances of same
driver class
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config.apt 88c1489
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 6, 2015, 4:20 p.m., Amareshwari Sriramadasu wrote:
> > lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java, line 38
> > <https://reviews.apache.org/r/39842/diff/1/?file=1113833#file1113833line38>
> >
> > Will it be possible to define driver types as constant strings or enum?
Can we not use an enum since it would imply only those drivres are supported. In theory anyone should be able to add a driver and should not need to modify the lens server code (Enum).If we use enum, a new driver type won't be part of it unless user modifies lens server code and will have to use constants till then.
Should I use constants then for test cases?
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review105437
-----------------------------------------------------------
On Nov. 20, 2015, 11:43 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2015, 11:43 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review105437
-----------------------------------------------------------
Overall changes looked clean and nice. I really enjoyed reviewing the changes :)
Looking forward for the updated patch with documentation and testcases added.
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java (line 38)
<https://reviews.apache.org/r/39842/#comment164038>
Will it be possible to define driver types as constant strings or enum?
- Amareshwari Sriramadasu
On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2015, 4:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107744
-----------------------------------------------------------
Seems http://lens.apache.org/lenshome/install-and-run.html requires update.
- Amareshwari Sriramadasu
On Nov. 24, 2015, 9:39 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 9:39 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107756
-----------------------------------------------------------
src/site/apt/admin/config-server.apt (line 42)
<https://reviews.apache.org/r/39842/#comment167039>
Conf/ should have been conf/
- Amareshwari Sriramadasu
On Nov. 24, 2015, 10:46 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 10:46 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> src/site/apt/lenshome/install-and-run.apt 9eadc5c
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107753
-----------------------------------------------------------
Ship it!
Ship It!
- Amareshwari Sriramadasu
On Nov. 24, 2015, 10:46 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 24, 2015, 10:46 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> src/site/apt/lenshome/install-and-run.apt 9eadc5c
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review108098
-----------------------------------------------------------
Ship it!
Ship It!
- Amareshwari Sriramadasu
On Nov. 26, 2015, 8:33 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 26, 2015, 8:33 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 9d8f198
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 49d62f9
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> src/site/apt/lenshome/install-and-run.apt 9eadc5c
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 27, 2015, 6:12 a.m., Rajat Khandelwal wrote:
> > lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml, lines 75-78
> > <https://reviews.apache.org/r/39842/diff/10/?file=1147794#file1147794line75>
> >
> > Do we need this for this test?
This is required for another test case . As of now we are using these two drivers only for TestQueryContraints. In future, if we decide to add them to other test cases , it ll be required .
I feel, No harm . Let all XMLs be similar for test.
> On Nov. 27, 2015, 6:12 a.m., Rajat Khandelwal wrote:
> > lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml, lines 80-83
> > <https://reviews.apache.org/r/39842/diff/10/?file=1147794#file1147794line80>
> >
> > Isn't name auto-inferred?
hmm. good question.
We added this property just to confirm if all drivers are loding specific configgration files in a test case. This was kinda the distinguishing property between all hive drivers. As of now we are using these two mock Hive drivers only for TestQueryContraints. In future, if we decide to add them to other test cases, it ll be required .
I feel, No harm . Let all XMLs be similar for test.
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review108191
-----------------------------------------------------------
On Nov. 27, 2015, 5:05 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2015, 5:05 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 19c4793
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 722a2da
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java cf795fa
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java 02b652e
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 7201e0d
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 9d8f198
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 8ba9353
> lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 7b6c560
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryConstraints.java eb94c89
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c8a1cc6
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mockHive/mockHive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> src/site/apt/lenshome/install-and-run.apt 9eadc5c
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review108191
-----------------------------------------------------------
lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml (lines 75 - 78)
<https://reviews.apache.org/r/39842/#comment167525>
Do we need this for this test?
lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml (lines 80 - 83)
<https://reviews.apache.org/r/39842/#comment167524>
Isn't name auto-inferred?
- Rajat Khandelwal
On Nov. 27, 2015, 10:35 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2015, 10:35 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 19c4793
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 722a2da
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java cf795fa
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java 02b652e
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 7201e0d
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 9d8f198
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 8ba9353
> lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 7b6c560
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryConstraints.java eb94c89
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c8a1cc6
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mockHive/mockHive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> src/site/apt/lenshome/install-and-run.apt 9eadc5c
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 27, 2015, 5:05 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
2 Chnages------
1. Patch Appy failed . Merged with latest code
2. Updated Test case TestQueryConstraints to use multiple drivers support . Also fixed a bug in this test case which was causing intermittent failure due to assertValidity() logic.
org.apache.lens.server.query.TestQueryConstraints.assertValidity()
if (count.running == 4) {
assertEquals(count.queued, 0);
} else {
assertEquals(count.waiting, 0);
}
Assuming … all candidate queries have moved to waiting from queued and 4 queries are running. Lets say the 4th query finishes before we call queryService.getQueryCountSnapshot() from the test case assertValidity(), so running queries =3. Now we call assertEquals(count.waiting, 0), but the query service thread has still not pushed a waiting query to launched . In this case we will see waiting queries >0 even though running queries != 4
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java 19c4793
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java 722a2da
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java cf795fa
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java 02b652e
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 7201e0d
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 9d8f198
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 8ba9353
lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 7b6c560
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryConstraints.java eb94c89
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c8a1cc6
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mockHive/mockHive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mockHive/mockHive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
src/site/apt/lenshome/install-and-run.apt 9eadc5c
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 26, 2015, 8:33 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
Fixed intermittent test failure in org.apache.lens.server.TestServerRestart.testHiveServerRestart()
- Updated thread safe type for LensSessionImpl.ResourceEntry.restoreCount
- Updated Test case
- Added Driver Qualified name to some some logs in HiveDriver and QueryExecutionServiceImpl
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 9d8f198
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/TestServerRestart.java 49d62f9
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
src/site/apt/lenshome/install-and-run.apt 9eadc5c
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 24, 2015, 11:28 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
"Conf" changed to "conf" in config-server.apt.
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
src/site/apt/lenshome/install-and-run.apt 9eadc5c
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 24, 2015, 10:46 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
updated documenatation
-install-and-run.apt
-config-server.apt
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
src/site/apt/lenshome/install-and-run.apt 9eadc5c
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 24, 2015, 9:39 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
- Updated test case to check for driver specific configration (lens.driver.test.drivername) also.
- Executed All test cases and tested lens examples
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 23, 2015, 1:05 p.m., Amareshwari Sriramadasu wrote:
> > tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml, line 31
> > <https://reviews.apache.org/r/39842/diff/5/?file=1135928#file1135928line31>
> >
> > Shall we add lens server db url with one more jdbc driver, by default?
This should be done in production envt where we decide on the lens server DB ( hsql or my sql /etc) ?
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107582
-----------------------------------------------------------
On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 11:40 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 23, 2015, 1:05 p.m., Amareshwari Sriramadasu wrote:
> > tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml, line 31
> > <https://reviews.apache.org/r/39842/diff/5/?file=1135928#file1135928line31>
> >
> > Shall we add lens server db url with one more jdbc driver, by default?
>
> Puneet Gupta wrote:
> This should be done in production envt where we decide on the lens server DB ( hsql or my sql /etc) ?
will take it as part of https://issues.apache.org/jira/browse/LENS-874
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107582
-----------------------------------------------------------
On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 11:40 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 23, 2015, 1:05 p.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java, line 231
> > <https://reviews.apache.org/r/39842/diff/5/?file=1135910#file1135910line231>
> >
> > Can we have one configuration to be different in all drivers and validate the conf value for each driver?
Added a test key "lens.driver.test.key2" . The value will be driver's qualified name . Validating the same in test case
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107582
-----------------------------------------------------------
On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 11:40 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107582
-----------------------------------------------------------
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java (line 231)
<https://reviews.apache.org/r/39842/#comment166804>
Can we have one configuration to be different in all drivers and validate the conf value for each driver?
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml (line 31)
<https://reviews.apache.org/r/39842/#comment166805>
Shall we add lens server db url with one more jdbc driver, by default?
- Amareshwari Sriramadasu
On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 11:40 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 23, 2015, 11:40 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
-Fixed Comments and updated documentation
-Added Driver Qualified name to logs
-Build and Test case sanity
-Started Server and Tested with Len-Examples
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.214s]
[INFO] Lens .............................................. SUCCESS [3.272s]
[INFO] Lens API .......................................... SUCCESS [24.859s]
[INFO] Lens API for server and extensions ................ SUCCESS [22.549s]
[INFO] Lens Cube ......................................... SUCCESS [5:34.207s]
[INFO] Lens DB storage ................................... SUCCESS [19.570s]
[INFO] Lens Query Library ................................ SUCCESS [15.746s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:55.797s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.876s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [16.485s]
[INFO] Lens Server ....................................... SUCCESS [8:14.493s]
[INFO] Lens client ....................................... SUCCESS [33.955s]
[INFO] Lens CLI .......................................... SUCCESS [53.948s]
[INFO] Lens Examples ..................................... SUCCESS [8.196s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.791s]
[INFO] Lens Distribution ................................. SUCCESS [9.214s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:20.010s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [2.412s]
[INFO] Lens Regression ................................... SUCCESS [13.762s]
[INFO] Lens UI ........................................... SUCCESS [27.177s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.464s
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 23, 2015, 10:44 a.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 340
> > <https://reviews.apache.org/r/39842/diff/4/?file=1134369#file1134369line340>
> >
> > I'm wondering whether we should move away from file names like `hivedriver-site.xml` and move to generic names like `driver-site.xml`. A driver's final conf will be an overlay of `drivers/lensdriver-site.xml`, `drivers/type/typedriver-site.xml`, `drivers/type/name/name-site.xml`. The conf initialization can be moved to `AbstractLensDriver` like with other common code. I'd tried to start a discussion on this in an earlier review.
>
> Amareshwari Sriramadasu wrote:
> Minimal changes would be always better. Current layout and changes look good to me. What do others think?
Common properties can be part of lens-site.xml for now and can be handled in AbstractLensDriver. If really required , later we can have a separate lensdriver-site.xml
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107553
-----------------------------------------------------------
On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 11:40 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Nov. 23, 2015, 10:44 a.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 340
> > <https://reviews.apache.org/r/39842/diff/4/?file=1134369#file1134369line340>
> >
> > I'm wondering whether we should move away from file names like `hivedriver-site.xml` and move to generic names like `driver-site.xml`. A driver's final conf will be an overlay of `drivers/lensdriver-site.xml`, `drivers/type/typedriver-site.xml`, `drivers/type/name/name-site.xml`. The conf initialization can be moved to `AbstractLensDriver` like with other common code. I'd tried to start a discussion on this in an earlier review.
Minimal changes would be always better. Current layout and changes look good to me. What do others think?
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107553
-----------------------------------------------------------
On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 11:40 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107553
-----------------------------------------------------------
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 340)
<https://reviews.apache.org/r/39842/#comment166780>
I'm wondering whether we should move away from file names like `hivedriver-site.xml` and move to generic names like `driver-site.xml`. A driver's final conf will be an overlay of `drivers/lensdriver-site.xml`, `drivers/type/typedriver-site.xml`, `drivers/type/name/name-site.xml`. The conf initialization can be moved to `AbstractLensDriver` like with other common code. I'd tried to start a discussion on this in an earlier review.
- Rajat Khandelwal
On Nov. 20, 2015, 5:13 p.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2015, 5:13 p.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 20, 2015, 12:06 p.m., Amareshwari Sriramadasu wrote:
> > lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java, line 438
> > <https://reviews.apache.org/r/39842/diff/4/?file=1134375#file1134375line438>
> >
> > Why is this test changed? Shouldnt have been.
The Driver config path was not set properly and the regular expression was not getting picked up which converts weekofyear to week.
Test Case Fixed ..
> On Nov. 20, 2015, 12:06 p.m., Amareshwari Sriramadasu wrote:
> > src/site/apt/admin/config-server.apt, line 45
> > <https://reviews.apache.org/r/39842/diff/4/?file=1134411#file1134411line45>
> >
> > Can you illustrate the conf directory layout here, as done review board description? You can put it as block instead of paragraph
Updated
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107352
-----------------------------------------------------------
On Nov. 20, 2015, 11:43 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2015, 11:43 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107352
-----------------------------------------------------------
Can you make sure you are able to bring up server in local setup and run examples?
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java (line 438)
<https://reviews.apache.org/r/39842/#comment166454>
Why is this test changed? Shouldnt have been.
src/site/apt/admin/config-server.apt (line 45)
<https://reviews.apache.org/r/39842/#comment166452>
Can you illustrate the conf directory layout here, as done review board description? You can put it as block instead of paragraph
- Amareshwari Sriramadasu
On Nov. 20, 2015, 11:43 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2015, 11:43 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 20, 2015, 11:43 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
There was an issue with preious two revisions . Please ignore them .
Updated Documentation, few comments and test case
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 20, 2015, 11:27 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Changes
-------
Added Documentation and test case. Removed redundant file (.gitignore)
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/
-----------------------------------------------------------
(Updated Nov. 20, 2015, 11:18 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
Bugs: LENS-123
https://issues.apache.org/jira/browse/LENS-123
Repository: lens
Description
-------
Ability to load multiple drivers on lens server.
As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
Made below changes in fix
1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
-conf
--drivers
---hive
----driver1
-----hivedriver-site.xml
----driver2
-----hivedriver-site.xml
---jdbc
----driver1
-----jdbcdriver-site.xml
----driver2
-----jdbcdriver-site.xml
Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
3. Updated driver interface
-configure method now takes driver name and type configuration parameters.
-added getFullyQualifiedName() method to identify the driver uniquely.
4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
<value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
8. Updated driver folder struture for test cases and lens server build (as per point1).
9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
Pending ( will upload this soon)
1. Adding test cases to test multiple drivers
2. updating documentation
Future Enhancements(that can be taken up as a separate JIRA)
1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
Diffs (updated)
-----
lens-api/.gitignore PRE-CREATION
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
lens-server/src/main/resources/lensserver-default.xml 5f268cb
lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
lens-server/src/test/resources/failing-query-driver-site.xml fee022d
lens-server/src/test/resources/hivedriver-site.xml f2aed88
lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
lens-server/src/test/resources/lens-site.xml cc887ef
src/site/apt/admin/config-server.apt 141f2b3
src/site/apt/admin/config.apt 88c1489
tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
tools/conf-pseudo-distr/server/lens-site.xml f43d07e
tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
tools/conf/server/hivedriver-site.xml 2e8e7fa
tools/conf/server/jdbcdriver-site.xml 37540dd
tools/conf/server/lens-site.xml 2b12b83
Diff: https://reviews.apache.org/r/39842/diff/
Testing
-------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
[INFO] Lens .............................................. SUCCESS [2.900s]
[INFO] Lens API .......................................... SUCCESS [25.172s]
[INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
[INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
[INFO] Lens DB storage ................................... SUCCESS [20.026s]
[INFO] Lens Query Library ................................ SUCCESS [15.835s]
[INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
[INFO] Lens Server ....................................... SUCCESS [8:17.418s]
[INFO] Lens client ....................................... SUCCESS [38.221s]
[INFO] Lens CLI .......................................... SUCCESS [56.723s]
[INFO] Lens Examples ..................................... SUCCESS [9.847s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
[INFO] Lens Distribution ................................. SUCCESS [9.897s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
[INFO] Lens Regression ................................... SUCCESS [12.622s]
[INFO] Lens UI ........................................... SUCCESS [26.741s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:00.146s
[INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
[INFO] Final Memory: 198M/2037M
[INFO] ------------------------------------------------------------------------
Tested lens-examples also after making the changes.
Thanks,
Puneet Gupta
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104876
-----------------------------------------------------------
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java (line 953)
<https://reviews.apache.org/r/39842/#comment163128>
This is an existing property which is set in lens-ctl. I added a constant to refer to it in code.
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java (line 195)
<https://reviews.apache.org/r/39842/#comment163130>
+1.
A test case should have pointed this too. Not sure why it didn't. Will change to lowercase.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java (line 423)
<https://reviews.apache.org/r/39842/#comment163129>
The condition which I followed for failing is when not even one driver is found. In other cases the query service can start.
Further check that created driver(s) can actaually execute queries (may be the database its pointing to is down or some configration parameter is wrong like user/pwd,etc )is not part of this change. It is kept same as before.
- Puneet Gupta
On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2015, 4:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 20, 2015, 7:52 a.m., Rajat Khandelwal wrote:
> > I see some instances where both the drivers are using same initialization but since property name is different(`lens.driver.hive.x` vs `lens.driver.jdbc.x`), the same code is repeated. Such code can go in the base class. example of such code:
> >
> > ```
> > this.queryConstraints = getImplementations(QUERY_LAUNCHING_CONSTRAINT_FACTORIES_KEY, this.conf);
> > this.selectionPolicies = getImplementations(WAITING_QUERIES_SELECTION_POLICY_FACTORIES_KEY, this.conf);
> > ```
> >
> > Secondly, the property names -- if unchanged -- will contain driver type now. I think that after this change, we can give a thought to property names. We can have a method in base class for looking up a property in conf:
> >
> >
> > ```
> > getProperty(conf, key){
> > return conf.get('lens.driver.name.x', conf.get('lens.driver.type.x', conf.get('x', DEFAULT)))
> > }
> > ```
> >
> > This will be backward compatible and allow us to move to a new convention of smaller property names.
> >
> > Thoughts?
>
> Amareshwari Sriramadasu wrote:
> If we have common properties they can simply be lens.driver.<property> (with out type) - each driver can give its own default value or override in site.
We can take this up as part of another JIRA, if really needed.
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107323
-----------------------------------------------------------
On Nov. 23, 2015, 11:40 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 11:40 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Nov. 20, 2015, 7:52 a.m., Rajat Khandelwal wrote:
> > I see some instances where both the drivers are using same initialization but since property name is different(`lens.driver.hive.x` vs `lens.driver.jdbc.x`), the same code is repeated. Such code can go in the base class. example of such code:
> >
> > ```
> > this.queryConstraints = getImplementations(QUERY_LAUNCHING_CONSTRAINT_FACTORIES_KEY, this.conf);
> > this.selectionPolicies = getImplementations(WAITING_QUERIES_SELECTION_POLICY_FACTORIES_KEY, this.conf);
> > ```
> >
> > Secondly, the property names -- if unchanged -- will contain driver type now. I think that after this change, we can give a thought to property names. We can have a method in base class for looking up a property in conf:
> >
> >
> > ```
> > getProperty(conf, key){
> > return conf.get('lens.driver.name.x', conf.get('lens.driver.type.x', conf.get('x', DEFAULT)))
> > }
> > ```
> >
> > This will be backward compatible and allow us to move to a new convention of smaller property names.
> >
> > Thoughts?
If we have common properties they can simply be lens.driver.<property> (with out type) - each driver can give its own default value or override in site.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107323
-----------------------------------------------------------
On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2015, 4:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review107323
-----------------------------------------------------------
I see some instances where both the drivers are using same initialization but since property name is different(`lens.driver.hive.x` vs `lens.driver.jdbc.x`), the same code is repeated. Such code can go in the base class. example of such code:
```
this.queryConstraints = getImplementations(QUERY_LAUNCHING_CONSTRAINT_FACTORIES_KEY, this.conf);
this.selectionPolicies = getImplementations(WAITING_QUERIES_SELECTION_POLICY_FACTORIES_KEY, this.conf);
```
Secondly, the property names -- if unchanged -- will contain driver type now. I think that after this change, we can give a thought to property names. We can have a method in base class for looking up a property in conf:
```
getProperty(conf, key){
return conf.get('lens.driver.name.x', conf.get('lens.driver.type.x', conf.get('x', DEFAULT)))
}
```
This will be backward compatible and allow us to move to a new convention of smaller property names.
Thoughts?
- Rajat Khandelwal
On Nov. 2, 2015, 9:38 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2015, 9:38 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java, line 100
> > <https://reviews.apache.org/r/39842/diff/1/?file=1113825#file1113825line100>
> >
> > Verify that no other occurences exist by doing `git grep selectedDriverClassName`
Found one instance in a comment.
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
-----------------------------------------------------------
On Nov. 20, 2015, 11:43 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2015, 11:43 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 83b919b
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java fdc8bfd
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/hive/hive2/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config-server.apt 141f2b3
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java, line 195
> > <https://reviews.apache.org/r/39842/diff/1/?file=1113859#file1113859line195>
> >
> > `driverName` or `drivername`?
>
> Puneet Gupta wrote:
> +1.
> A test case should have pointed this too. Not sure why it didn't. Will change to lowercase.
Most DBs(incl.. MySQL) do not require strict case for column names. Thats why the test cases are passing, I guess.
Will change to lower case though.
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
-----------------------------------------------------------
On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2015, 4:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java, line 953
> > <https://reviews.apache.org/r/39842/diff/1/?file=1113848#file1113848line953>
> >
> > Should this be lens.server.config.location?
This is an existing property which is set in lens-ctl. I added a constant to refer to it in code.
> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java, line 195
> > <https://reviews.apache.org/r/39842/diff/1/?file=1113859#file1113859line195>
> >
> > `driverName` or `drivername`?
+1.
A test case should have pointed this too. Not sure why it didn't. Will change to lowercase.
> On Nov. 2, 2015, 10:35 a.m., Rajat Khandelwal wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java, line 439
> > <https://reviews.apache.org/r/39842/diff/1/?file=1113860#file1113860line439>
> >
> > Should we throw an error here? It'll be better to fail fast.
The condition which I followed for failing is when not even one driver is found. In other cases (provided driver configartion is correct - as in configuration is syntactically correct, drivers folder exists , driver class instance creation is successful) the query service can start.
Further check that created driver(s) can actaually execute queries (may be the database its pointing to is down or some configration parameter is wrong like user/pwd,etc )is not part of this change. It is kept same as before.
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
-----------------------------------------------------------
On Nov. 2, 2015, 4:08 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2015, 4:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>
Re: Review Request 39842: Ability to load different instances of same
driver class
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39842/#review104688
-----------------------------------------------------------
lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java (line 100)
<https://reviews.apache.org/r/39842/#comment162937>
Verify that no other occurences exist by doing `git grep selectedDriverClassName`
lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java (line 953)
<https://reviews.apache.org/r/39842/#comment162945>
Should this be lens.server.config.location?
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java (line 195)
<https://reviews.apache.org/r/39842/#comment162943>
`driverName` or `drivername`?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java (line 423)
<https://reviews.apache.org/r/39842/#comment162944>
Should we throw an error here? It'll be better to fail fast.
- Rajat Khandelwal
On Nov. 2, 2015, 9:38 a.m., Puneet Gupta wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39842/
> -----------------------------------------------------------
>
> (Updated Nov. 2, 2015, 9:38 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajat Khandelwal.
>
>
> Bugs: LENS-123
> https://issues.apache.org/jira/browse/LENS-123
>
>
> Repository: lens
>
>
> Description
> -------
>
> Ability to load multiple drivers on lens server.
>
> As of now only one driver instance of each type (hive,jdbc,es,etc) can be loaded by lens server. Hence lens can not support for example, two jdbc instances, one for MySql and one for Vertica or just two different MySQl deployments. This can be a big limitation for some deployments.
>
> Made below changes in fix
> 1. Based on the approaches discussed in JIRA(LENS-123), the below folder structure was picked. (Please refer to the discussion on JIRA if required - last few comments )
> -conf
> --drivers
> ---hive
> ----driver1
> -----hivedriver-site.xml
> ----driver2
> -----hivedriver-site.xml
> ---jdbc
> ----driver1
> -----jdbcdriver-site.xml
> ----driver2
> -----jdbcdriver-site.xml
> Note: drivers configuration is read from "drivers" directory under conf location. Conf loaction is set while starting the server as -Dconfig.location = "<conf path>"
>
> 2. Added an abstract class "org.apache.lens.server.api.driver.AbstractLensDriver". This has common implemenation for getting lens driver's fully qualified name and some methods for getting driver specific resource paths. All existing driver implementations extend this class. In future, we can also add common functionality here(if required) without affecting existing driver implementations that extend this abstract class.
>
> 3. Updated driver interface
> -configure method now takes driver name and type configuration parameters.
> -added getFullyQualifiedName() method to identify the driver uniquely.
>
> 4. Value of lens.server.drivers property in lens-site.xml (and default-lens-site) updated to include driver type and driver class name (earlier only driver class name was present).
> <value>hive:org.apache.lens.driver.hive.HiveDriver,jdbc:org.apache.lens.driver.jdbc.JDBCDriver</value>
>
> 5. Updated LensServerDAO to use driver qualified name instead of driver class name. The table column name is chnaged to drivername form driverclass
>
> 6. The driver instance was earlier referred to by its class name (Example:org.apache.lens.driver.hive.HiveDriver) everywhere in code. This is now changed to driver's fully qualified name which has the driver type and driver name (Example :hive/hive1)
>
> 7. All metrics were also relying on driver class name. This is also changed to driver's fully qualified name.
>
> 8. Updated driver folder struture for test cases and lens server build (as per point1).
>
> 9. Query execution service will fail to start if no drivers are found. (This is a change from previous flow . We can discuss this if req.)
>
>
> Pending ( will upload this soon)
> 1. Adding test cases to test multiple drivers
> 2. updating documentation
>
> Future Enhancements(that can be taken up as a separate JIRA)
> 1.Multiple versions of a datasource to be supported ( say Hive and Hive on spark use different versions of Hive. This will not work as of now as we have common Hive jars as part of war classpath. We need to have driver specific jars which can have different version. For this, the jars and other hive version specific resources should be picked from driver's folder by a separate class loader).
>
>
> Diffs
> -----
>
> lens-api/src/main/java/org/apache/lens/api/query/LensPreparedQuery.java 9595ce9
> lens-api/src/main/java/org/apache/lens/api/query/LensQuery.java 204ecee
> lens-cli/src/main/java/org/apache/lens/cli/commands/LensQueryCommands.java 096fd7a
> lens-cli/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-client/src/main/java/org/apache/lens/client/LensStatement.java 0a511f0
> lens-client/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-cube/src/test/java/org/apache/lens/driver/cube/TestMinCostSelector.java 72f1497
> lens-driver-es/src/main/java/org/apache/lens/driver/es/ESDriver.java 14d9f99
> lens-driver-es/src/test/java/org/apache/lens/driver/es/ESDriverTest.java f453416
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java fc57c94
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java 98edc28
> lens-driver-hive/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-driver-hive/src/test/resources/hivedriver-site.xml 613938d
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java a8b980f
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 7772d16
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 053e20d
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 425bd6f
> lens-driver-jdbc/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-driver-jdbc/src/test/resources/jdbcdriver-site.xml 1202074
> lens-ml-lib/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-ml-lib/src/test/resources/lens-site.xml 3d5dbef
> lens-query-lib/src/test/java/org/apache/lens/lib/query/TestAbstractFileFormatter.java 40e1cdc
> lens-server-api/src/main/java/org/apache/lens/server/api/LensConfConstants.java 7ee0749
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/AbstractLensDriver.java PRE-CREATION
> lens-server-api/src/main/java/org/apache/lens/server/api/driver/LensDriver.java a5a60d7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/AbstractQueryContext.java 0c980a2
> lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java feac938
> lens-server-api/src/main/java/org/apache/lens/server/api/query/FinishedLensQuery.java 89053aa
> lens-server-api/src/main/java/org/apache/lens/server/api/query/PreparedQueryContext.java b6f669b
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java 9b491d1
> lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java 2d86589
> lens-server-api/src/test/java/org/apache/lens/server/api/query/MockQueryContext.java fd6b560
> lens-server-api/src/test/java/org/apache/lens/server/api/query/TestAbstractQueryContext.java e41f2f4
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java b9dd286
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 4d8ae51
> lens-server/src/main/java/org/apache/lens/server/rewrite/RewriteUtil.java 6c464fb
> lens-server/src/main/resources/lensserver-default.xml 5f268cb
> lens-server/src/test/java/org/apache/lens/server/LensJerseyTest.java 3dad050
> lens-server/src/test/java/org/apache/lens/server/query/TestEventService.java 1dab35e
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java bc1463f
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
> lens-server/src/test/java/org/apache/lens/server/rewrite/TestRewriting.java 2827b96
> lens-server/src/test/resources/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> lens-server/src/test/resources/drivers/mock/fail1/failing-query-driver-site.xml PRE-CREATION
> lens-server/src/test/resources/failing-query-driver-site.xml fee022d
> lens-server/src/test/resources/hivedriver-site.xml f2aed88
> lens-server/src/test/resources/jdbcdriver-site.xml 1b14f54
> lens-server/src/test/resources/lens-site.xml cc887ef
> src/site/apt/admin/config.apt 88c1489
> tools/conf-pseudo-distr/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf-pseudo-distr/server/hivedriver-site.xml 4804356
> tools/conf-pseudo-distr/server/jdbcdriver-site.xml 37540dd
> tools/conf-pseudo-distr/server/lens-site.xml f43d07e
> tools/conf/server/drivers/hive/hive1/hivedriver-site.xml PRE-CREATION
> tools/conf/server/drivers/jdbc/jdbc1/jdbcdriver-site.xml PRE-CREATION
> tools/conf/server/hivedriver-site.xml 2e8e7fa
> tools/conf/server/jdbcdriver-site.xml 37540dd
> tools/conf/server/lens-site.xml 2b12b83
>
> Diff: https://reviews.apache.org/r/39842/diff/
>
>
> Testing
> -------
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.090s]
> [INFO] Lens .............................................. SUCCESS [2.900s]
> [INFO] Lens API .......................................... SUCCESS [25.172s]
> [INFO] Lens API for server and extensions ................ SUCCESS [24.540s]
> [INFO] Lens Cube ......................................... SUCCESS [5:17.973s]
> [INFO] Lens DB storage ................................... SUCCESS [20.026s]
> [INFO] Lens Query Library ................................ SUCCESS [15.835s]
> [INFO] Lens Hive Driver .................................. SUCCESS [2:54.352s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [39.054s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [19.714s]
> [INFO] Lens Server ....................................... SUCCESS [8:17.418s]
> [INFO] Lens client ....................................... SUCCESS [38.221s]
> [INFO] Lens CLI .......................................... SUCCESS [56.723s]
> [INFO] Lens Examples ..................................... SUCCESS [9.847s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [1.488s]
> [INFO] Lens Distribution ................................. SUCCESS [9.897s]
> [INFO] Lens ML Lib ....................................... SUCCESS [1:22.714s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.841s]
> [INFO] Lens Regression ................................... SUCCESS [12.622s]
> [INFO] Lens UI ........................................... SUCCESS [26.741s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 23:00.146s
> [INFO] Finished at: Mon Nov 02 03:14:06 UTC 2015
> [INFO] Final Memory: 198M/2037M
> [INFO] ------------------------------------------------------------------------
>
> Tested lens-examples also after making the changes.
>
>
> Thanks,
>
> Puneet Gupta
>
>