You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Ankit Kailaswar <an...@gmail.com> on 2018/12/12 08:21:30 UTC
Review Request 69554: Lens HA changes
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Diffs
-----
lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java a0d8e0ba
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java ccd2a3b7
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 7d81375c
Diff: https://reviews.apache.org/r/69554/diff/1/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
> > Lines 617 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line617>
> >
> > can this be moved to the if condition? this would ensure there is atleast one row
we should rely on this to throw exception. If resultset has no rows or for any other type of error or exception we are relying on finsihed_query table to fetch query details.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
> > Lines 623 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line623>
> >
> > Can this be shifted to final block? Also DBUtils.close(stmt) and DBUtils.close(rs) makes more sense. This applies here and everywhere else
DBUtils.close(stmt) and DBUtils.close(rs) throws SQLException, not a good idea to have it in final block.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
> > Line 390 (original), 827 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line827>
> >
> > while all of existing code in lensserverdao is using commons db utils, any particular reason we are using vanilla java code in the new code? to maintain consistency in the code, it seems better to use the db utils library to define the new apis as well, IMO
db utils api sunc as DBRunner that we are using currently perofrms one to one serialisaition between java class and mysql table. In case of query context class and LensSession.PersistInfo it will add unnecessary complexity of modifying these table definition as and when we introduce new filed in these classes.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 3680 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line3680>
> >
> > Why can't we put this code in existing validatesession in Baselensservice than defining a new api altogether?
>
> Ankit Kailaswar wrote:
> moved.
SessionValidator interface is better place for this api, moved it there.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
> > Line 579 (original), 616 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line616>
> >
> > While all the write operations on sessions are handled in this code, I am not able to see any changes in list and get kind of apis in this class
> > For instance, what happens if getAllSessionParameters api gets called with a session id from other lens instance, in which case I assume , restoresessionfrom db needs to be called in the getSession api of BaseLensService, which doesn't seem to be handled here
> > restoresessionfromdb might have to be called in other apis involving write operations as well, which I am unable to see
we are calling it for
1. requests in query resource
2. get/set params , list/add resources in session.
3. all requests for scheduler service
4. all requests for metastore service.
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211269
-----------------------------------------------------------
On Dec. 18, 2018, 12:20 p.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2018, 12:20 p.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java cb29c9ea
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/scheduler/ScheduleResource.java 1d5959b2
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
>
>
> Diff: https://reviews.apache.org/r/69554/diff/5/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
> > Lines 590 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line590>
> >
> > Instead of making a db call for every session , we need to have some batching mechanism for better performance
>
> Ankit Kailaswar wrote:
> This has to be instantaneous. If we batch then we might end up in situation where session is closed on one server but it is still present on another.
>
> Rajitha R wrote:
> By batching I mean to say if we can send a list of ids to delete at once, than one of them each time.
I am assuming here you it is w.r.t. session clenaup thread only. for other scenario it has to be instantaneous. In case of session cleanup If we create a batch it will add bit of complexity here. Consider a scenario where query is failing because it has one or two session which are not in db then none of the session will be cleaned up it means none of the session will be cleaned up from db as well as in memory, we can increment counter though but this will add dependency of adderesing the issue more quickly since session backlog will increase at higher rate.
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211269
-----------------------------------------------------------
On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2019, 11:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Build Log :
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
> [INFO] Lens .............................................. SUCCESS [5.788s]
> [INFO] Lens API .......................................... SUCCESS [48.711s]
> [INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
> [INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
> [INFO] Lens DB storage ................................... SUCCESS [29.279s]
> [INFO] Lens Query Library ................................ SUCCESS [22.507s]
> [INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
> [INFO] Lens Server ....................................... SUCCESS [12:19.915s]
> [INFO] Lens client ....................................... SUCCESS [1:19.368s]
> [INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
> [INFO] Lens Examples ..................................... SUCCESS [7.287s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
> [INFO] Lens Distribution ................................. SUCCESS [10.625s]
> [INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
> [INFO] Lens Regression ................................... SUCCESS [9.296s]
> [INFO] Lens UI ........................................... SUCCESS [28.750s]
> [INFO] Lens Contrib ...................................... SUCCESS [0.229s]
> [INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
> [INFO] Lens Python Client ................................ SUCCESS [0.231s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 30:34.060s
> [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
> [INFO] Final Memory: 300M/3097M
> [INFO] ------------------------------------------------------------------------
> [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
> + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
> 0
>
>
> Diffs
> -----
>
> lens-client/src/test/java/org/apache/lens/client/TestLensClient.java d9e60fb7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
>
>
> Diff: https://reviews.apache.org/r/69554/diff/8/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 2258 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line2258>
> >
> > makes more sense to move this to all executequery apis than in createcontext api to be in sync with the api definition
make more sense, moved.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 2367 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line2367>
> >
> > How is this supposed to work?
> > Given the scenario where Lens 2 is given a query Q1 from Lens 1, the db may still contain an older status which it could have updated before going down
> > Who exactly is updating the latest status of Q1 in the db?
only lens server 1 is updating status. Lens 2 will just return its status from db. lens1 will keep on updating status till it goes down. Lens 2 will always return last updated status in db.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 3680 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113076#file2113076line3680>
> >
> > Why can't we put this code in existing validatesession in Baselensservice than defining a new api altogether?
moved.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
> > Lines 431 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line431>
> >
> > can't see any calls to this api anywhere
> > Also from our offline discussion, from my understanding , both instances share a separate persistent state(on hdfs) and they read session state from db only when it can't find a given session id in its memory
> > Why does it require changes in existing restoresession in the first place?
removed, This is needed for phase2.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
> > Lines 590 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113078#file2113078line590>
> >
> > Instead of making a db call for every session , we need to have some batching mechanism for better performance
This has to be instantaneous. If we batch then we might end up in situation where session is closed on one server but it is still present on another.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java
> > Line 615 (original), 616 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113079#file2113079line616>
> >
> > public interface Externalizable extends Serializable {
removed.
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211269
-----------------------------------------------------------
On Dec. 12, 2018, 8:21 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2018, 8:21 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java a0d8e0ba
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java ccd2a3b7
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 7d81375c
>
>
> Diff: https://reviews.apache.org/r/69554/diff/1/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Dec. 13, 2018, 4:54 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
> > Lines 623 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113075#file2113075line623>
> >
> > Can this be shifted to final block? Also DBUtils.close(stmt) and DBUtils.close(rs) makes more sense. This applies here and everywhere else
>
> Ankit Kailaswar wrote:
> DBUtils.close(stmt) and DBUtils.close(rs) throws SQLException, not a good idea to have it in final block.
>
> Rajitha R wrote:
> You could use closeQuietly in that case
replaced.
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211269
-----------------------------------------------------------
On Dec. 24, 2018, 10:50 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 24, 2018, 10:50 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
>
>
> Diff: https://reviews.apache.org/r/69554/diff/6/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Rajitha R <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211269
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
Lines 617 (patched)
<https://reviews.apache.org/r/69554/#comment296215>
can this be moved to the if condition? this would ensure there is atleast one row
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
Lines 623 (patched)
<https://reviews.apache.org/r/69554/#comment296216>
Can this be shifted to final block? Also DBUtils.close(stmt) and DBUtils.close(rs) makes more sense. This applies here and everywhere else
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
Line 390 (original), 827 (patched)
<https://reviews.apache.org/r/69554/#comment296217>
while all of existing code in lensserverdao is using commons db utils, any particular reason we are using vanilla java code in the new code? to maintain consistency in the code, it seems better to use the db utils library to define the new apis as well, IMO
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Lines 2258 (patched)
<https://reviews.apache.org/r/69554/#comment296222>
makes more sense to move this to all executequery apis than in createcontext api to be in sync with the api definition
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Lines 2367 (patched)
<https://reviews.apache.org/r/69554/#comment296223>
How is this supposed to work?
Given the scenario where Lens 2 is given a query Q1 from Lens 1, the db may still contain an older status which it could have updated before going down
Who exactly is updating the latest status of Q1 in the db?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Lines 3680 (patched)
<https://reviews.apache.org/r/69554/#comment296224>
Why can't we put this code in existing validatesession in Baselensservice than defining a new api altogether?
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 431 (patched)
<https://reviews.apache.org/r/69554/#comment296220>
can't see any calls to this api anywhere
Also from our offline discussion, from my understanding , both instances share a separate persistent state(on hdfs) and they read session state from db only when it can't find a given session id in its memory
Why does it require changes in existing restoresession in the first place?
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 590 (patched)
<https://reviews.apache.org/r/69554/#comment296219>
Instead of making a db call for every session , we need to have some batching mechanism for better performance
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Line 579 (original), 616 (patched)
<https://reviews.apache.org/r/69554/#comment296221>
While all the write operations on sessions are handled in this code, I am not able to see any changes in list and get kind of apis in this class
For instance, what happens if getAllSessionParameters api gets called with a session id from other lens instance, in which case I assume , restoresessionfrom db needs to be called in the getSession api of BaseLensService, which doesn't seem to be handled here
restoresessionfromdb might have to be called in other apis involving write operations as well, which I am unable to see
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java
Line 615 (original), 616 (patched)
<https://reviews.apache.org/r/69554/#comment296218>
public interface Externalizable extends Serializable {
- Rajitha R
On Dec. 12, 2018, 8:21 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2018, 8:21 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java a0d8e0ba
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java ccd2a3b7
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 7d81375c
>
>
> Diff: https://reviews.apache.org/r/69554/diff/1/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java
> > Line 54 (original), 55 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113070#file2113070line55>
> >
> > AbstractQueryContext already implements Serialzable
missed that, reverted.
> On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java
> > Lines 69 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113072#file2113072line69>
> >
> > A new implementation for existing restoresession would make more sense than adding new api
yes, make sense, reverted.
> On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
> > Line 24 (original)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113073#file2113073line24>
> >
> > can this be reverted ?
reverted
> On Dec. 12, 2018, 3:38 p.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
> > Line 583 (original), 580 (patched)
> > <https://reviews.apache.org/r/69554/diff/1/?file=2113073#file2113073line584>
> >
> > "should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server"
> > The above statement says to validate session from mysql, but here its checking inmemory and throwing the exception
we are validating session from memory map as well as from mysql. (please check getSession() definition).
Current flow is,
1. check if it is present in memory first.
2. If it is present in memory then validate if its mark for expired or not.
3. If it is not present in memory then validate if its presnet in mysql, if it it presnet in mysql then validate if its mark for closed or not.
4. If it is not present in mysql too then throw an exception for sesion not present.
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211230
-----------------------------------------------------------
On Dec. 18, 2018, 12:20 p.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2018, 12:20 p.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java cb29c9ea
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/scheduler/ScheduleResource.java 1d5959b2
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
>
>
> Diff: https://reviews.apache.org/r/69554/diff/5/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Rajitha R <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211230
-----------------------------------------------------------
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java
Line 54 (original), 55 (patched)
<https://reviews.apache.org/r/69554/#comment296171>
AbstractQueryContext already implements Serialzable
lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java
Lines 69 (patched)
<https://reviews.apache.org/r/69554/#comment296172>
A new implementation for existing restoresession would make more sense than adding new api
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Line 24 (original)
<https://reviews.apache.org/r/69554/#comment296173>
can this be reverted ?
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Line 583 (original), 580 (patched)
<https://reviews.apache.org/r/69554/#comment296174>
"should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server"
The above statement says to validate session from mysql, but here its checking inmemory and throwing the exception
- Rajitha R
On Dec. 12, 2018, 8:21 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 12, 2018, 8:21 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java a0d8e0ba
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server-api/src/main/java/org/apache/lens/server/api/session/SessionService.java ccd2a3b7
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 7d81375c
>
>
> Diff: https://reviews.apache.org/r/69554/diff/1/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote:
> > lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java
> > Lines 31 (patched)
> > <https://reviews.apache.org/r/69554/diff/5/?file=2114506#file2114506line31>
> >
> > Can we not retain the same name validateSession?
changed name
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
> > Line 392 (original), 401 (patched)
> > <https://reviews.apache.org/r/69554/diff/5/?file=2114508#file2114508line401>
> >
> > validatesession not being called here
getsession is internal call. We have to validate user session in mysql only for user requests and not for internal session calls. There wont be a scenario where getsession is called and session is not present in memory because it will be created itself on user's first request only. Also on other hand if we try to call validatesession here then it will be called for lot of time in query prep/execute/submission phases. Not an good idea. That could be the reason it was not in place for older implementation as well.
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
> > Lines 607 (patched)
> > <https://reviews.apache.org/r/69554/diff/5/?file=2114508#file2114508line608>
> >
> > Any particular reason we are throwing e here and not daoE? May be throw e should be outside the if condition.
need to throw exception on condition here, corrected.
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
> > Lines 611 (patched)
> > <https://reviews.apache.org/r/69554/diff/5/?file=2114508#file2114508line612>
> >
> > can validateSession and validateSessionId api's be swapped here such that no changes will be required where validateSession api is called?
changed name
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 1175 (patched)
> > <https://reviews.apache.org/r/69554/diff/5/?file=2114512#file2114512line1175>
> >
> > I am afraid if these continuous db updates will slow down the system
This update happens 3-4 times in query lifecycle, once for each state change (SUBMITTED, QUEUED, RUNNING, FAILED). Considering query execution time this should not slow down the system.
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Line 1541 (original), 1566 (patched)
> > <https://reviews.apache.org/r/69554/diff/5/?file=2114512#file2114512line1566>
> >
> > why is db update statement NOT called outside the switch statement and only in EXECUTED case ?
it has to be for all, moved.
> On Dec. 21, 2018, 6:10 a.m., Rajitha R wrote:
> > lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
> > Lines 538 (patched)
> > <https://reviews.apache.org/r/69554/diff/5/?file=2114515#file2114515line540>
> >
> > Why is this required? Can we not implement these counters in the existing way by calling notifyevent?
I cant see any event implementation for metric service in code also we are using metric service directly to handle metric counters at all places in code. Correct me if I am wrong.
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211463
-----------------------------------------------------------
On Jan. 8, 2019, 10:52 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2019, 10:52 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
>
>
> Diff: https://reviews.apache.org/r/69554/diff/7/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Rajitha R <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211463
-----------------------------------------------------------
lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java
Lines 31 (patched)
<https://reviews.apache.org/r/69554/#comment296617>
Can we not retain the same name validateSession?
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Line 392 (original), 401 (patched)
<https://reviews.apache.org/r/69554/#comment296673>
validatesession not being called here
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Lines 604 (patched)
<https://reviews.apache.org/r/69554/#comment296669>
possibility of npe
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Lines 604 (patched)
<https://reviews.apache.org/r/69554/#comment296670>
no null check on persistinfo, possibility of npe
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Lines 607 (patched)
<https://reviews.apache.org/r/69554/#comment296671>
Any particular reason we are throwing e here and not daoE? May be throw e should be outside the if condition.
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Lines 611 (patched)
<https://reviews.apache.org/r/69554/#comment296668>
can validateSession and validateSessionId api's be swapped here such that no changes will be required where validateSession api is called?
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
Lines 501 (patched)
<https://reviews.apache.org/r/69554/#comment296676>
Can we have same naming convention for better clarity? we can have insertActionSession like insertActiveQuery instead of insertActionSessions , same applies to update
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java
Lines 701 (patched)
<https://reviews.apache.org/r/69554/#comment296618>
These should be moved to final block?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Lines 142 (patched)
<https://reviews.apache.org/r/69554/#comment296621>
can we have a more intuitive name like? ACTIVE_QUERY_INSERT_ERROR_COUNTER
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Lines 1175 (patched)
<https://reviews.apache.org/r/69554/#comment296672>
I am afraid if these continuous db updates will slow down the system
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Line 1541 (original), 1566 (patched)
<https://reviews.apache.org/r/69554/#comment296620>
why is db update statement NOT called outside the switch statement and only in EXECUTED case ?
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 429 (patched)
<https://reviews.apache.org/r/69554/#comment296674>
can this be reverted?
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 538 (patched)
<https://reviews.apache.org/r/69554/#comment296675>
Why is this required? Can we not implement these counters in the existing way by calling notifyevent?
- Rajitha R
On Dec. 18, 2018, 12:20 p.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2018, 12:20 p.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java cb29c9ea
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/scheduler/ScheduleResource.java 1d5959b2
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
>
>
> Diff: https://reviews.apache.org/r/69554/diff/5/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Jan. 7, 2019, 3:47 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 1176 (patched)
> > <https://reviews.apache.org/r/69554/diff/6/?file=2116214#file2116214line1176>
> >
> > no counter for failures?
added
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211710
-----------------------------------------------------------
On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2019, 11:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Build Log :
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
> [INFO] Lens .............................................. SUCCESS [5.788s]
> [INFO] Lens API .......................................... SUCCESS [48.711s]
> [INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
> [INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
> [INFO] Lens DB storage ................................... SUCCESS [29.279s]
> [INFO] Lens Query Library ................................ SUCCESS [22.507s]
> [INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
> [INFO] Lens Server ....................................... SUCCESS [12:19.915s]
> [INFO] Lens client ....................................... SUCCESS [1:19.368s]
> [INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
> [INFO] Lens Examples ..................................... SUCCESS [7.287s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
> [INFO] Lens Distribution ................................. SUCCESS [10.625s]
> [INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
> [INFO] Lens Regression ................................... SUCCESS [9.296s]
> [INFO] Lens UI ........................................... SUCCESS [28.750s]
> [INFO] Lens Contrib ...................................... SUCCESS [0.229s]
> [INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
> [INFO] Lens Python Client ................................ SUCCESS [0.231s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 30:34.060s
> [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
> [INFO] Final Memory: 300M/3097M
> [INFO] ------------------------------------------------------------------------
> [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
> + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
> 0
>
>
> Diffs
> -----
>
> lens-client/src/test/java/org/apache/lens/client/TestLensClient.java d9e60fb7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
>
>
> Diff: https://reviews.apache.org/r/69554/diff/8/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211710
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Lines 133 (patched)
<https://reviews.apache.org/r/69554/#comment297232>
Why is DAO created for every lens service?
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Lines 1176 (patched)
<https://reviews.apache.org/r/69554/#comment297233>
no counter for failures?
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 91 (patched)
<https://reviews.apache.org/r/69554/#comment297234>
declaration required here? Can you mark with required scope - private or protected?
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 397 (patched)
<https://reviews.apache.org/r/69554/#comment297235>
Is initialized in BaseLensService as well.
- Amareshwari Sriramadasu
On Dec. 24, 2018, 10:50 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 24, 2018, 10:50 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
>
>
> Diff: https://reviews.apache.org/r/69554/diff/6/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review212235
-----------------------------------------------------------
lens-client/src/test/java/org/apache/lens/client/TestLensClient.java
Line 292 (original)
<https://reviews.apache.org/r/69554/#comment297958>
Why is fail assert removed? No more socket exception expected? If so, whats up with catch block below?
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Line 571 (original), 577 (patched)
<https://reviews.apache.org/r/69554/#comment297959>
why is method renamed to get, which is not returning any session here?
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 112 (patched)
<https://reviews.apache.org/r/69554/#comment297960>
remove commented code?
- Amareshwari Sriramadasu
On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2019, 11:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Build Log :
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
> [INFO] Lens .............................................. SUCCESS [5.788s]
> [INFO] Lens API .......................................... SUCCESS [48.711s]
> [INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
> [INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
> [INFO] Lens DB storage ................................... SUCCESS [29.279s]
> [INFO] Lens Query Library ................................ SUCCESS [22.507s]
> [INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
> [INFO] Lens Server ....................................... SUCCESS [12:19.915s]
> [INFO] Lens client ....................................... SUCCESS [1:19.368s]
> [INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
> [INFO] Lens Examples ..................................... SUCCESS [7.287s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
> [INFO] Lens Distribution ................................. SUCCESS [10.625s]
> [INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
> [INFO] Lens Regression ................................... SUCCESS [9.296s]
> [INFO] Lens UI ........................................... SUCCESS [28.750s]
> [INFO] Lens Contrib ...................................... SUCCESS [0.229s]
> [INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
> [INFO] Lens Python Client ................................ SUCCESS [0.231s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 30:34.060s
> [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
> [INFO] Final Memory: 300M/3097M
> [INFO] ------------------------------------------------------------------------
> [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
> + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
> 0
>
>
> Diffs
> -----
>
> lens-client/src/test/java/org/apache/lens/client/TestLensClient.java d9e60fb7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
>
>
> Diff: https://reviews.apache.org/r/69554/diff/8/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review212304
-----------------------------------------------------------
Ship it!
Ship It!
- Amareshwari Sriramadasu
On Jan. 24, 2019, 11:32 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2019, 11:32 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Build Log :
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
> [INFO] Lens .............................................. SUCCESS [5.788s]
> [INFO] Lens API .......................................... SUCCESS [48.711s]
> [INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
> [INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
> [INFO] Lens DB storage ................................... SUCCESS [29.279s]
> [INFO] Lens Query Library ................................ SUCCESS [22.507s]
> [INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
> [INFO] Lens Server ....................................... SUCCESS [12:19.915s]
> [INFO] Lens client ....................................... SUCCESS [1:19.368s]
> [INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
> [INFO] Lens Examples ..................................... SUCCESS [7.287s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
> [INFO] Lens Distribution ................................. SUCCESS [10.625s]
> [INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
> [INFO] Lens Regression ................................... SUCCESS [9.296s]
> [INFO] Lens UI ........................................... SUCCESS [28.750s]
> [INFO] Lens Contrib ...................................... SUCCESS [0.229s]
> [INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
> [INFO] Lens Python Client ................................ SUCCESS [0.231s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 30:34.060s
> [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
> [INFO] Final Memory: 300M/3097M
> [INFO] ------------------------------------------------------------------------
> [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
> + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
> 0
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 74806afe
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
>
>
> Diff: https://reviews.apache.org/r/69554/diff/9/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Jan. 28, 2019, 7:01 p.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Build Log :
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
[INFO] Lens .............................................. SUCCESS [5.788s]
[INFO] Lens API .......................................... SUCCESS [48.711s]
[INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
[INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
[INFO] Lens DB storage ................................... SUCCESS [29.279s]
[INFO] Lens Query Library ................................ SUCCESS [22.507s]
[INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
[INFO] Lens Server ....................................... SUCCESS [12:19.915s]
[INFO] Lens client ....................................... SUCCESS [1:19.368s]
[INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
[INFO] Lens Examples ..................................... SUCCESS [7.287s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
[INFO] Lens Distribution ................................. SUCCESS [10.625s]
[INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
[INFO] Lens Regression ................................... SUCCESS [9.296s]
[INFO] Lens UI ........................................... SUCCESS [28.750s]
[INFO] Lens Contrib ...................................... SUCCESS [0.229s]
[INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
[INFO] Lens Python Client ................................ SUCCESS [0.231s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30:34.060s
[INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
[INFO] Final Memory: 300M/3097M
[INFO] ------------------------------------------------------------------------
[Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
+ ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
0
Diffs (updated)
-----
lens-client/src/test/java/org/apache/lens/server/MockQueryExecutionServiceImpl.java 9b55fb62
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 74806afe
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
Diff: https://reviews.apache.org/r/69554/diff/10/
Changes: https://reviews.apache.org/r/69554/diff/9-10/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Jan. 24, 2019, 11:32 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Build Log :
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
[INFO] Lens .............................................. SUCCESS [5.788s]
[INFO] Lens API .......................................... SUCCESS [48.711s]
[INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
[INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
[INFO] Lens DB storage ................................... SUCCESS [29.279s]
[INFO] Lens Query Library ................................ SUCCESS [22.507s]
[INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
[INFO] Lens Server ....................................... SUCCESS [12:19.915s]
[INFO] Lens client ....................................... SUCCESS [1:19.368s]
[INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
[INFO] Lens Examples ..................................... SUCCESS [7.287s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
[INFO] Lens Distribution ................................. SUCCESS [10.625s]
[INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
[INFO] Lens Regression ................................... SUCCESS [9.296s]
[INFO] Lens UI ........................................... SUCCESS [28.750s]
[INFO] Lens Contrib ...................................... SUCCESS [0.229s]
[INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
[INFO] Lens Python Client ................................ SUCCESS [0.231s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30:34.060s
[INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
[INFO] Final Memory: 300M/3097M
[INFO] ------------------------------------------------------------------------
[Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
+ ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
0
Diffs (updated)
-----
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java 74806afe
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
Diff: https://reviews.apache.org/r/69554/diff/9/
Changes: https://reviews.apache.org/r/69554/diff/8-9/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Jan. 22, 2019, 11:08 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description (updated)
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Build Log :
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
[INFO] Lens .............................................. SUCCESS [5.788s]
[INFO] Lens API .......................................... SUCCESS [48.711s]
[INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
[INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
[INFO] Lens DB storage ................................... SUCCESS [29.279s]
[INFO] Lens Query Library ................................ SUCCESS [22.507s]
[INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
[INFO] Lens Server ....................................... SUCCESS [12:19.915s]
[INFO] Lens client ....................................... SUCCESS [1:19.368s]
[INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
[INFO] Lens Examples ..................................... SUCCESS [7.287s]
[INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
[INFO] Lens Distribution ................................. SUCCESS [10.625s]
[INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
[INFO] Lens Regression ................................... SUCCESS [9.296s]
[INFO] Lens UI ........................................... SUCCESS [28.750s]
[INFO] Lens Contrib ...................................... SUCCESS [0.229s]
[INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
[INFO] Lens Python Client ................................ SUCCESS [0.231s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30:34.060s
[INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
[INFO] Final Memory: 300M/3097M
[INFO] ------------------------------------------------------------------------
[Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
+ ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
0
Diffs (updated)
-----
lens-client/src/test/java/org/apache/lens/client/TestLensClient.java d9e60fb7
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
Diff: https://reviews.apache.org/r/69554/diff/8/
Changes: https://reviews.apache.org/r/69554/diff/7-8/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211719
-----------------------------------------------------------
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java
Lines 133 (patched)
<https://reviews.apache.org/r/69554/#comment297271>
made it static final.
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
Lines 1176 (patched)
<https://reviews.apache.org/r/69554/#comment297274>
We have a counters for insert and delete. update will be intermidiate operation between these 2 calls though of not having one more extra counter. Added though.
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 91 (patched)
<https://reviews.apache.org/r/69554/#comment297272>
this is not required here, can use base class instance. removed as member variable.
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java
Lines 397 (patched)
<https://reviews.apache.org/r/69554/#comment297273>
removed
- Ankit Kailaswar
On Jan. 8, 2019, 10:52 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Jan. 8, 2019, 10:52 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
>
>
> Diff: https://reviews.apache.org/r/69554/diff/7/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Jan. 8, 2019, 10:52 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Diffs (updated)
-----
lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
Diff: https://reviews.apache.org/r/69554/diff/7/
Changes: https://reviews.apache.org/r/69554/diff/6-7/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
> On Jan. 7, 2019, 3:48 a.m., Amareshwari Sriramadasu wrote:
> > No unit tests added for the change introduced. Can you add unit tests?
added test cases for lensserver dao functions.
- Ankit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211711
-----------------------------------------------------------
On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2019, 11:08 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Build Log :
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [2.740s]
> [INFO] Lens .............................................. SUCCESS [5.788s]
> [INFO] Lens API .......................................... SUCCESS [48.711s]
> [INFO] Lens API for server and extensions ................ SUCCESS [33.537s]
> [INFO] Lens Cube ......................................... SUCCESS [7:43.672s]
> [INFO] Lens DB storage ................................... SUCCESS [29.279s]
> [INFO] Lens Query Library ................................ SUCCESS [22.507s]
> [INFO] Lens Hive Driver .................................. SUCCESS [1:27.344s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [1:24.416s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [44.067s]
> [INFO] Lens Server ....................................... SUCCESS [12:19.915s]
> [INFO] Lens client ....................................... SUCCESS [1:19.368s]
> [INFO] Lens CLI .......................................... SUCCESS [1:19.672s]
> [INFO] Lens Examples ..................................... SUCCESS [7.287s]
> [INFO] Lens Ship Jars to Distributed Cache ............... SUCCESS [0.661s]
> [INFO] Lens Distribution ................................. SUCCESS [10.625s]
> [INFO] Lens ML Lib ....................................... SUCCESS [53.247s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [1.418s]
> [INFO] Lens Regression ................................... SUCCESS [9.296s]
> [INFO] Lens UI ........................................... SUCCESS [28.750s]
> [INFO] Lens Contrib ...................................... SUCCESS [0.229s]
> [INFO] Lens Contributed Clients .......................... SUCCESS [0.226s]
> [INFO] Lens Python Client ................................ SUCCESS [0.231s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 30:34.060s
> [INFO] Finished at: Thu Jan 17 10:47:01 UTC 2019
> [INFO] Final Memory: 300M/3097M
> [INFO] ------------------------------------------------------------------------
> [Platform_Common_Job] $ /bin/sh -xe /tmp/jenkins5693507200862759570.sh
> + ruby /d0/jenkins/scripts/licenseCheck.rb /d0/jenkins/workspace/Platform_Common_Job
> 0
>
>
> Diffs
> -----
>
> lens-client/src/test/java/org/apache/lens/client/TestLensClient.java d9e60fb7
> lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
> lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java PRE-CREATION
> lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java 066525b7
>
>
> Diff: https://reviews.apache.org/r/69554/diff/8/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211711
-----------------------------------------------------------
No unit tests added for the change introduced. Can you add unit tests?
- Amareshwari Sriramadasu
On Dec. 24, 2018, 10:50 a.m., Ankit Kailaswar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> -----------------------------------------------------------
>
> (Updated Dec. 24, 2018, 10:50 a.m.)
>
>
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
>
>
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
>
>
> Repository: lens
>
>
> Description
> -------
>
> Implementation
>
> Session
>
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
> For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
>
> if
>
> session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
>
> else If
>
> session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
>
> else if
>
> session is not present in mysql and in server’s in memory map then return “invalid session”.
>
> Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
>
>
>
> Query
>
> For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
> For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query plan/estimate.
>
>
>
> Issue addressed
>
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
>
>
> Diffs
> -----
>
> lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
> lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
> lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
> lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
> lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
> lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
> lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
>
>
> Diff: https://reviews.apache.org/r/69554/diff/6/
>
>
> Testing
> -------
>
> Unit testing
>
>
> Thanks,
>
> Ankit Kailaswar
>
>
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Dec. 24, 2018, 10:50 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Diffs (updated)
-----
lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
Diff: https://reviews.apache.org/r/69554/diff/6/
Changes: https://reviews.apache.org/r/69554/diff/5-6/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Dec. 18, 2018, 12:20 p.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Diffs (updated)
-----
lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java cb29c9ea
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/scheduler/ScheduleResource.java 1d5959b2
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
Diff: https://reviews.apache.org/r/69554/diff/5/
Changes: https://reviews.apache.org/r/69554/diff/4-5/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Dec. 18, 2018, 12:10 p.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Diffs (updated)
-----
lens-server-api/src/main/java/org/apache/lens/server/api/SessionValidator.java 65403191
lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
lens-server/src/main/java/org/apache/lens/server/metastore/MetastoreResource.java cb29c9ea
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/scheduler/ScheduleResource.java 1d5959b2
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
Diff: https://reviews.apache.org/r/69554/diff/4/
Changes: https://reviews.apache.org/r/69554/diff/3-4/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Dec. 14, 2018, 12:06 p.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Diffs (updated)
-----
lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
src/site/apt/admin/config.apt 4cee5ae2
Diff: https://reviews.apache.org/r/69554/diff/3/
Changes: https://reviews.apache.org/r/69554/diff/2-3/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar
Re: Review Request 69554: Lens HA changes
Posted by Ankit Kailaswar <an...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
-----------------------------------------------------------
(Updated Dec. 14, 2018, 11:42 a.m.)
Review request for lens, Amareshwari Sriramadasu and Rajitha R.
Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538
Repository: lens
Description
-------
Implementation
Session
We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql server. persisting entire session object is not required. We need to do this as first thing while creating user session.
For each request from client lens server should not validate session id from its in memory map. It should rather validate it from persisted session in mysql server.
if
session is present in mysql and not present in lens server’s in memory map(session manager) then it should create a lens session with same session id and add it to in memory map
else If
session is not present in mysql and present in server’s in memory map then it signifies that user have initiated close session from other host and we will need to remove this session from current host’s in memory map as well
else if
session is not present in mysql and in server’s in memory map then return “invalid session”.
Whenever session expired/closed then we need to remove this session from lens server’s in memory map and mysql.
Query
For sync queries lens server will close connection as soon as it is stopped or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of rescheduling all queries which were in running or queued state at the time of restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 then user should be able to poll on query status. Lens server will first check query id in its inmemory maps if it is not present then it will check in “finished_query” table in mysql else in “current_query” table in mysql. In this case we will continue showing old status of query since it is scheduled by host1. As soon as host1 comes up it will reschedule these queries and will change the status. Optionally we can have host2 to move these queries in “allqueries” map of its query service which will take care of recovering, in this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query plan/estimate.
Issue addressed
Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.
Diffs (updated)
-----
lens-server-api/src/main/java/org/apache/lens/server/api/events/AsyncEventListener.java 94734658
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryContext.java a0d8e0ba
lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java a8031098
lens-server/src/main/java/org/apache/lens/server/BaseLensService.java 9364872d
lens-server/src/main/java/org/apache/lens/server/error/LensServerErrorCode.java ef43371a
lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java cc6ca7d4
lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java 07a2107a
lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java 47b40a8f
lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java f6d43d70
lens-server/src/main/java/org/apache/lens/server/session/LensSessionImpl.java 7d81375c
src/site/apt/admin/config.apt 4cee5ae2
Diff: https://reviews.apache.org/r/69554/diff/2/
Changes: https://reviews.apache.org/r/69554/diff/1-2/
Testing
-------
Unit testing
Thanks,
Ankit Kailaswar