You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Sushil Mohanty <su...@gmail.com> on 2016/05/25 07:15:03 UTC
Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR for
JDBC queries due to connection unavailability
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/
-----------------------------------------------------------
Review request for lens.
Bugs: LENS-1155
https://issues.apache.org/jira/browse/LENS-1155
Repository: lens
Description
-------
Issue :
When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
Fix :
Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
Diffs
-----
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 4eee354
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java b2319a9
lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
Diff: https://reviews.apache.org/r/47820/diff/
Testing
-------
Thanks,
Sushil Mohanty
Re: Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR
for JDBC queries due to connection unavailability
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/#review134742
-----------------------------------------------------------
Ship it!
LGTM
- Amareshwari Sriramadasu
On May 25, 2016, 9:01 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47820/
> -----------------------------------------------------------
>
> (Updated May 25, 2016, 9:01 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1155
> https://issues.apache.org/jira/browse/LENS-1155
>
>
> Repository: lens
>
>
> Description
> -------
>
> Issue :
> When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
>
> Fix :
> Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
> lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
> lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
>
> Diff: https://reviews.apache.org/r/47820/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR
for JDBC queries due to connection unavailability
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/
-----------------------------------------------------------
(Updated May 25, 2016, 9:43 a.m.)
Review request for lens.
Changes
-------
updated test result.
Bugs: LENS-1155
https://issues.apache.org/jira/browse/LENS-1155
Repository: lens
Description
-------
Issue :
When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
Fix :
Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
Diffs
-----
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
Diff: https://reviews.apache.org/r/47820/diff/
Testing (updated)
-------
mvn clean install.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [1.568s]
[INFO] Lens .............................................. SUCCESS [3.851s]
[INFO] Lens API .......................................... SUCCESS [21.238s]
[INFO] Lens API for server and extensions ................ SUCCESS [17.135s]
[INFO] Lens Cube ......................................... SUCCESS [14:05.321s]
[INFO] Lens DB storage ................................... SUCCESS [17.297s]
[INFO] Lens Query Library ................................ SUCCESS [14.356s]
[INFO] Lens Hive Driver .................................. SUCCESS [1:22.104s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [37.757s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [22.355s]
[INFO] Lens Server ....................................... SUCCESS [9:17.492s]
[INFO] Lens client ....................................... SUCCESS [1:31.286s]
[INFO] Lens CLI .......................................... SUCCESS [1:18.286s]
[INFO] Lens Examples ..................................... SUCCESS [7.670s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.624s]
[INFO] Lens Distribution ................................. SUCCESS [9.221s]
[INFO] Lens ML Lib ....................................... SUCCESS [1:07.127s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.984s]
[INFO] Lens Regression ................................... SUCCESS [11.468s]
[INFO] Lens UI ........................................... SUCCESS [25.865s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:54.713s
[INFO] Finished at: Wed May 25 09:26:53 UTC 2016
[INFO] Final Memory: 201M/2506M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR
for JDBC queries due to connection unavailability
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/
-----------------------------------------------------------
(Updated May 25, 2016, 9:01 a.m.)
Review request for lens.
Changes
-------
Addressed review comments.
Bugs: LENS-1155
https://issues.apache.org/jira/browse/LENS-1155
Repository: lens
Description
-------
Issue :
When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
Fix :
Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
Diffs (updated)
-----
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
Diff: https://reviews.apache.org/r/47820/diff/
Testing
-------
Thanks,
Sushil Mohanty
Re: Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR
for JDBC queries due to connection unavailability
Posted by Sushil Mohanty <su...@gmail.com>.
> On May 25, 2016, 8:20 a.m., Puneet Gupta wrote:
> > lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java, line 79
> > <https://reviews.apache.org/r/47820/diff/1/?file=1393290#file1393290line79>
> >
> > Do we need this extra line ?
Added for some testing. Not required, removed.
> On May 25, 2016, 8:20 a.m., Puneet Gupta wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java, line 46
> > <https://reviews.apache.org/r/47820/diff/1/?file=1393292#file1393292line46>
> >
> > This change is required?
No. Put it in wrong place. It shoud be in JDBCDriver init method.
- Sushil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/#review134723
-----------------------------------------------------------
On May 25, 2016, 9:01 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47820/
> -----------------------------------------------------------
>
> (Updated May 25, 2016, 9:01 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1155
> https://issues.apache.org/jira/browse/LENS-1155
>
>
> Repository: lens
>
>
> Description
> -------
>
> Issue :
> When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
>
> Fix :
> Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
> lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
> lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
>
> Diff: https://reviews.apache.org/r/47820/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR
for JDBC queries due to connection unavailability
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/#review134723
-----------------------------------------------------------
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java (line 79)
<https://reviews.apache.org/r/47820/#comment199617>
Do we need this extra line ?
lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java (line 46)
<https://reviews.apache.org/r/47820/#comment199618>
This change is required?
- Puneet Gupta
On May 25, 2016, 7:15 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47820/
> -----------------------------------------------------------
>
> (Updated May 25, 2016, 7:15 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1155
> https://issues.apache.org/jira/browse/LENS-1155
>
>
> Repository: lens
>
>
> Description
> -------
>
> Issue :
> When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
>
> Fix :
> Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
> lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 4eee354
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
> lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java b2319a9
> lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
>
> Diff: https://reviews.apache.org/r/47820/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR
for JDBC queries due to connection unavailability
Posted by Sushil Mohanty <su...@gmail.com>.
> On May 25, 2016, 8:34 a.m., Amareshwari Sriramadasu wrote:
> > lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java, line 41
> > <https://reviews.apache.org/r/47820/diff/1/?file=1393287#file1393287line41>
> >
> > Can there be a case where selectedDriver is not JDBCDriver ?
> >
> > then we are actually returning false.
Don't think any such case.
- Sushil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/#review134729
-----------------------------------------------------------
On May 25, 2016, 9:01 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47820/
> -----------------------------------------------------------
>
> (Updated May 25, 2016, 9:01 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1155
> https://issues.apache.org/jira/browse/LENS-1155
>
>
> Repository: lens
>
>
> Description
> -------
>
> Issue :
> When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
>
> Fix :
> Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
> lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
> lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
>
> Diff: https://reviews.apache.org/r/47820/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 47820: LENS-1155 : Seeing INTERNAL_SERVER_ERROR
for JDBC queries due to connection unavailability
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47820/#review134729
-----------------------------------------------------------
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java (line 41)
<https://reviews.apache.org/r/47820/#comment199623>
Can there be a case where selectedDriver is not JDBCDriver ?
then we are actually returning false.
lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java (line 46)
<https://reviews.apache.org/r/47820/#comment199624>
Seems wrong
- Amareshwari Sriramadasu
On May 25, 2016, 7:15 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47820/
> -----------------------------------------------------------
>
> (Updated May 25, 2016, 7:15 a.m.)
>
>
> Review request for lens.
>
>
> Bugs: LENS-1155
> https://issues.apache.org/jira/browse/LENS-1155
>
>
> Repository: lens
>
>
> Description
> -------
>
> Issue :
> When user launching InMemory Queries, the connection is released back to pool only after result set is read completely or after query is purged. For InMemory Queries (not persisted by lens server), the query state is changed to SUCCESSFUL even before connection is released.This causes connection pool to throw error for new queries whenever JDBC driver is running at almost full load.
>
> Fix :
> Added a new constraint at JDBC driver to check max allowed connections. At any given time to launch a query the number of query context objects held by driver should be less than connection pool max size.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/JDBCDriver.java 044a19c
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraint.java PRE-CREATION
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/MaxJDBCConnectionCheckConstraintFactory.java PRE-CREATION
> lens-driver-jdbc/src/main/resources/jdbcdriver-default.xml b5b6164
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 4eee354
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJdbcDriver.java 67f6c1f
> lens-server-api/src/main/java/org/apache/lens/server/api/query/constraint/MaxConcurrentDriverQueriesConstraint.java b2319a9
> lens-server-api/src/main/java/org/apache/lens/server/api/util/LensUtil.java a0691a5
>
> Diff: https://reviews.apache.org/r/47820/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sushil Mohanty
>
>