You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Deepak Barr <de...@gmail.com> on 2015/11/03 19:43:06 UTC
Review Request 39911: LENS-760 : Session close should not result in
running query failures
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/
-----------------------------------------------------------
Review request for lens.
Repository: lens
Description
-------
OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
Diffs
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
Diff: https://reviews.apache.org/r/39911/diff/
Testing
-------
Yes.
1. Created session.
2. Fired a query.
3. Deleted the session before query finishes.
4. Query keeps running and finishes eventually.
Thanks,
Deepak Barr
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 4, 2015, 3:51 p.m., Puneet Gupta wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1249
> > <https://reviews.apache.org/r/39911/diff/1/?file=1114914#file1114914line1249>
> >
> > What happens to skipped(open) sessions ? do we need to close them later once query execution is over ?
>
> Deepak Barr wrote:
> I think we can rely of hive server to clean it up eventually.
Not sure if that will be OK. Can someone comment form Hive perspective..
Will lot open(skipped) hive sessions form lens create issues while creating new Hive sessions?
What is the timeout for Hive sessions in production?
Incase its Not Ok form Hive prespective, we might have to inititiate periodic cleanup on lens side. (If not periodic, it can also be triggeregd when any query is purged. Not very relaible though)
Only other problem would be server restart. Where do we store the open hive sessions details ? We might have to serailize that info too.
We an take this up separately also if required and close this issue.
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review105069
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Nov. 4, 2015, 3:51 p.m., Puneet Gupta wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1249
> > <https://reviews.apache.org/r/39911/diff/1/?file=1114914#file1114914line1249>
> >
> > What happens to skipped(open) sessions ? do we need to close them later once query execution is over ?
I think we can rely of hive server to clean it up eventually.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review105069
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Nov. 4, 2015, 3:51 p.m., Puneet Gupta wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1249
> > <https://reviews.apache.org/r/39911/diff/1/?file=1114914#file1114914line1249>
> >
> > What happens to skipped(open) sessions ? do we need to close them later once query execution is over ?
>
> Deepak Barr wrote:
> I think we can rely of hive server to clean it up eventually.
>
> Puneet Gupta wrote:
> Not sure if that will be OK. Can someone comment form Hive perspective..
> Will lot open(skipped) hive sessions form lens create issues while creating new Hive sessions?
> What is the timeout for Hive sessions in production?
>
> Incase its Not Ok form Hive prespective, we might have to inititiate periodic cleanup on lens side. (If not periodic, it can also be triggeregd when any query is purged. Not very relaible though)
> Only other problem would be server restart. Where do we store the open hive sessions details ? We might have to serailize that info too.
>
> We an take this up separately also if required and close this issue.
Hive does not do any cleanup of sessions. This will lead to session leaks and Memory pile up on hive. We should take care of closing the hive sessions when all its queries are completed.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review105069
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review105069
-----------------------------------------------------------
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 516)
<https://reviews.apache.org/r/39911/#comment163446>
Should this be done in case on Execptions also, as in should we do this in finally clause ?
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 1249)
<https://reviews.apache.org/r/39911/#comment163451>
What happens to skipped(open) sessions ? do we need to close them later once query execution is over ?
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 1254)
<https://reviews.apache.org/r/39911/#comment163450>
Can we print session id also here?
- Puneet Gupta
On Nov. 3, 2015, 6:43 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2015, 6:43 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Dec. 10, 2015, 4:41 a.m., Puneet Gupta wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 535
> > <https://reviews.apache.org/r/39911/diff/3/?file=1150389#file1150389line535>
> >
> > If closeQuery() is not called before this, which happens for (result == null || !(result instanceof HiveInMemoryResultSet), how will the oprphan session be closed?
> > As of now closeQuery() is doing that also.
Looking at the execute method and its comment[assuming this is only called for executing explain/insert/set/delete/etc... queries which don't ask to fetch data.),I think this method is only called for queries that do not return result,so I assumed closeQuery() will be called.Looking at the code, I can see this method is called for "add jar" and "explain" queries.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109474
-----------------------------------------------------------
On Dec. 2, 2015, 10:40 a.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2015, 10:40 a.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Dec. 10, 2015, 4:41 a.m., Puneet Gupta wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, lines 803-804
> > <https://reviews.apache.org/r/39911/diff/3/?file=1150389#file1150389line803>
> >
> > Please log the closing of session .
Not required in latest diff. Session close on driver close is removed.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109474
-----------------------------------------------------------
On Dec. 10, 2015, 10:19 a.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2015, 10:19 a.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Puneet Gupta <pu...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109474
-----------------------------------------------------------
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 535)
<https://reviews.apache.org/r/39911/#comment169058>
If closeQuery() is not called before this, which happens for (result == null || !(result instanceof HiveInMemoryResultSet), how will the oprphan session be closed?
As of now closeQuery() is doing that also.
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (lines 803 - 804)
<https://reviews.apache.org/r/39911/#comment169050>
Please log the closing of session .
- Puneet Gupta
On Dec. 2, 2015, 10:40 a.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2015, 10:40 a.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Dec. 7, 2015, 7:56 a.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 803
> > <https://reviews.apache.org/r/39911/diff/2-3/?file=1119315#file1119315line803>
> >
> > If sessions are getting closed at query finish, there shouldn't be any orphans now. Let's log the orphans and the cause(if possible) for their being orphans.
>
> Deepak Barr wrote:
> The close() method is never called except for some test classes. So I am not quite sure if there will be orphan sessions. For example, if a query is running on a orphaned session and driver.close() is called, then we would want to close the orphaned sessions ( which will also kill the query running on that session)
Agree with Deepak here. driver.close should not close the sessions, and i think right now, driver.close itself is not called. We can remove session close from here to be consistent with semantics.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109130
-----------------------------------------------------------
On Dec. 2, 2015, 10:40 a.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2015, 10:40 a.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Dec. 7, 2015, 7:56 a.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 803
> > <https://reviews.apache.org/r/39911/diff/2-3/?file=1119315#file1119315line803>
> >
> > If sessions are getting closed at query finish, there shouldn't be any orphans now. Let's log the orphans and the cause(if possible) for their being orphans.
The close() method is never called except for some test classes. So I am not quite sure if there will be orphan sessions. For example, if a query is running on a orphaned session and driver.close() is called, then we would want to close the orphaned sessions ( which will also kill the query running on that session)
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109130
-----------------------------------------------------------
On Dec. 2, 2015, 10:40 a.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2015, 10:40 a.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109130
-----------------------------------------------------------
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 802)
<https://reviews.apache.org/r/39911/#comment168561>
If sessions are getting closed at query finish, there shouldn't be any orphans now. Let's log the orphans and the cause(if possible) for their being orphans.
- Rajat Khandelwal
On Dec. 2, 2015, 4:10 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 2, 2015, 4:10 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/
-----------------------------------------------------------
(Updated Dec. 11, 2015, 2:48 p.m.)
Review request for lens.
Changes
-------
updated with reactor summary
Repository: lens
Description
-------
OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
Diffs
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
Diff: https://reviews.apache.org/r/39911/diff/
Testing (updated)
-------
Yes.
1. Created session.
2. Fired a query.
3. Deleted the session before query finishes.
4. Query keeps running and finishes eventually.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .............................. SUCCESS [ 2.491 s]
[INFO] Lens ............................................... SUCCESS [ 13.332 s]
[INFO] Lens API ........................................... SUCCESS [ 31.967 s]
[INFO] Lens API for server and extensions ................. SUCCESS [ 16.379 s]
[INFO] Lens Cube .......................................... SUCCESS [08:22 min]
[INFO] Lens DB storage .................................... SUCCESS [ 28.926 s]
[INFO] Lens Query Library ................................. SUCCESS [ 11.509 s]
[INFO] Lens Hive Driver ................................... SUCCESS [02:22 min]
[INFO] Lens Driver for JDBC ............................... SUCCESS [ 43.142 s]
[INFO] Lens Elastic Search Driver ......................... SUCCESS [ 14.213 s]
[INFO] Lens Server ........................................ SUCCESS [13:01 min]
[INFO] Lens client ........................................ SUCCESS [01:07 min]
[INFO] Lens CLI ........................................... SUCCESS [01:35 min]
[INFO] Lens Examples ...................................... SUCCESS [ 12.293 s]
[INFO] Lens Ship Jars to Distributed Cache ................ SUCCESS [ 1.499 s]
[INFO] Lens Distribution .................................. SUCCESS [ 18.882 s]
[INFO] Lens ML Lib ........................................ SUCCESS [02:32 min]
[INFO] Lens ML Ext Distribution ........................... SUCCESS [ 3.597 s]
[INFO] Lens Regression .................................... SUCCESS [ 15.574 s]
[INFO] Lens UI ............................................ SUCCESS [ 7.101 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 32:44 min
[INFO] Finished at: 2015-12-11T18:23:29+05:30
[INFO] Final Memory: 143M/869M
[INFO] ------------------------------------------------------------------------
Thanks,
Deepak Barr
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109959
-----------------------------------------------------------
Ship it!
Please update testing done with latest patch.
- Amareshwari Sriramadasu
On Dec. 11, 2015, 12:55 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 11, 2015, 12:55 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/
-----------------------------------------------------------
(Updated Dec. 11, 2015, 12:55 p.m.)
Review request for lens.
Changes
-------
Added persistence for opHandleToSessionMap.
Repository: lens
Description
-------
OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
Diffs (updated)
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
Diff: https://reviews.apache.org/r/39911/diff/
Testing
-------
Yes.
1. Created session.
2. Fired a query.
3. Deleted the session before query finishes.
4. Query keeps running and finishes eventually.
Thanks,
Deepak Barr
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Dec. 11, 2015, 7:11 a.m., Amareshwari Sriramadasu wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1295
> > <https://reviews.apache.org/r/39911/diff/4/?file=1158468#file1158468line1295>
> >
> > Should we not persist opHandleToSessionHandle map as well? Otherwise, orphaned sessions will be still orphaned upon lens server restart.
I agree. My bad.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109927
-----------------------------------------------------------
On Dec. 11, 2015, 12:55 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 11, 2015, 12:55 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review109927
-----------------------------------------------------------
Deepak, can you mark the review issues as fixed/dropped accordingly ?
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 1284)
<https://reviews.apache.org/r/39911/#comment169646>
Should we not persist opHandleToSessionHandle map as well? Otherwise, orphaned sessions will be still orphaned upon lens server restart.
- Amareshwari Sriramadasu
On Dec. 10, 2015, 10:19 a.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2015, 10:19 a.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
> lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/
-----------------------------------------------------------
(Updated Dec. 10, 2015, 10:19 a.m.)
Review request for lens.
Changes
-------
removed session close logic from driver.close().
Repository: lens
Description
-------
OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
Diffs (updated)
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
Diff: https://reviews.apache.org/r/39911/diff/
Testing
-------
Yes.
1. Created session.
2. Fired a query.
3. Deleted the session before query finishes.
4. Query keeps running and finishes eventually.
Thanks,
Deepak Barr
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/
-----------------------------------------------------------
(Updated Dec. 2, 2015, 10:40 a.m.)
Review request for lens.
Changes
-------
Proper clean up of orphaned hive sessions
Repository: lens
Description
-------
OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
Diffs (updated)
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java a84c679
lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestRemoteHiveDriver.java ab5ada9
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java f6693aa
Diff: https://reviews.apache.org/r/39911/diff/
Testing
-------
Yes.
1. Created session.
2. Fired a query.
3. Deleted the session before query finishes.
4. Query keeps running and finishes eventually.
Thanks,
Deepak Barr
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Nov. 9, 2015, 12:57 p.m., Rajat Khandelwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1260
> > <https://reviews.apache.org/r/39911/diff/2/?file=1119315#file1119315line1260>
> >
> > Add test cases for not closeable?
didn't get you ? I added a test case that closes the lens session but wont close hivesession because of running query.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review105671
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review105671
-----------------------------------------------------------
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 1260)
<https://reviews.apache.org/r/39911/#comment164275>
Add test cases for not closeable?
- Rajat Khandelwal
On Nov. 9, 2015, 12:28 a.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 9, 2015, 12:28 a.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Nov. 26, 2015, 9:14 a.m., Amareshwari Sriramadasu wrote:
> > Deepak Barr, Is there any update on this for review comments?
Sorry, I was occupied with something else. Will resume work on this. From the review comments, I am thinking we should do the following -
1. For sync queries (CLI), closing the session will kill the queries . For async, queries wont be killed.
2. We will clean up the orphaned hive sessions periodically when the queries on the hive session finishes. I imagined hive might clean up the session after certain timeout but the functionaliy is available version 0.14 onwards (HIVE-5799).
Let me know if its fine with everyone.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review108103
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Nov. 26, 2015, 9:14 a.m., Amareshwari Sriramadasu wrote:
> > Deepak Barr, Is there any update on this for review comments?
>
> Deepak Barr wrote:
> Sorry, I was occupied with something else. Will resume work on this. From the review comments, I am thinking we should do the following -
>
> 1. For sync queries (CLI), closing the session will kill the queries . For async, queries wont be killed.
> 2. We will clean up the orphaned hive sessions periodically when the queries on the hive session finishes. I imagined hive might clean up the session after certain timeout but the functionaliy is available version 0.14 onwards (HIVE-5799).
>
> Let me know if its fine with everyone.
Seems fine. please go ahead and update.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review108103
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review108103
-----------------------------------------------------------
Deepak Barr, Is there any update on this for review comments?
- Amareshwari Sriramadasu
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Nov. 13, 2015, 6:57 a.m., Pranav Agarwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1245
> > <https://reviews.apache.org/r/39911/diff/2/?file=1119315#file1119315line1245>
> >
> > In my opinion we shouldn't change the default behavior of this API. Since the user asked to close the session this semantic implies that the underying queries should be killed.
> >
> > If we want to address the usecase of closing the session that are inactive and are not running any queries then driver should have an API to return the running queries for a session. And the caller should make a call whether they want to first check if there are any running queries in the driver before calling closeSession.
>
> Puneet Gupta wrote:
> I was wondering when user has issued an "asynchronous" query, why should the user wait till the query finishes (sometimes more than a day). He can close the cli/session and come back later and check the status or kill the query or do other operations available via query handle. Keeping the session open when a user doesn't intend to do any further oprations other than just waiting for the async query result, can be wastage of resources for lens server. When a sync query is issued, the behaviour can be different, if really required.
>
> Pranav Agarwal wrote:
> In my opinion - If the queries are running, even after explictly closing the session then it's counter intutive. Today the user know's if the session is closed then all the queries fired from that session will be killed and that makes good sense to me.
>
> I do understand the issue about async long running queries to continue even after the session is closed. To handle that we can probably accept an additonal parameter say "noHangUp" at the time of query submission and the sessions running queries with noHangUp shouldn't be closed in Hive but can be removed from LensServer. I would think only users querying via CLI will need this option, as the session close for rest of the apps can be managed by the client code.
>> In my opinion we shouldn't change the default behavior of this API. Since the user asked to close the session this semantic implies that the underying queries should be killed.
I would say this is not the default behavior, it right now fails only for few queries, which have pending tasks.
I propose we should have the following behavior, if everyone agrees :
- All lens queries are async queries. Any query submitted through REST api is accessible in other sessions as well. so, session close here should not kill the query.
- CLI provides a way to submit sync queries, by waiting for completion of them on the same submission; such queries should be killed by cli on cli session close/control+c etc. Queries submitted through CLI through --async option should be accessible across sessions. With existing option of --async and sync way, i dont think we should add --noHandup as another option.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review106370
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Pranav Agarwal <pr...@gmail.com>.
> On Nov. 13, 2015, 6:57 a.m., Pranav Agarwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1245
> > <https://reviews.apache.org/r/39911/diff/2/?file=1119315#file1119315line1245>
> >
> > In my opinion we shouldn't change the default behavior of this API. Since the user asked to close the session this semantic implies that the underying queries should be killed.
> >
> > If we want to address the usecase of closing the session that are inactive and are not running any queries then driver should have an API to return the running queries for a session. And the caller should make a call whether they want to first check if there are any running queries in the driver before calling closeSession.
>
> Puneet Gupta wrote:
> I was wondering when user has issued an "asynchronous" query, why should the user wait till the query finishes (sometimes more than a day). He can close the cli/session and come back later and check the status or kill the query or do other operations available via query handle. Keeping the session open when a user doesn't intend to do any further oprations other than just waiting for the async query result, can be wastage of resources for lens server. When a sync query is issued, the behaviour can be different, if really required.
In my opinion - If the queries are running, even after explictly closing the session then it's counter intutive. Today the user know's if the session is closed then all the queries fired from that session will be killed and that makes good sense to me.
I do understand the issue about async long running queries to continue even after the session is closed. To handle that we can probably accept an additonal parameter say "noHangUp" at the time of query submission and the sessions running queries with noHangUp shouldn't be closed in Hive but can be removed from LensServer. I would think only users querying via CLI will need this option, as the session close for rest of the apps can be managed by the client code.
- Pranav
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review106370
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
> On Nov. 13, 2015, 6:57 a.m., Pranav Agarwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1258
> > <https://reviews.apache.org/r/39911/diff/2/?file=1119315#file1119315line1258>
> >
> > If we indeed decide to change the semantic of this method then we should minimally communicate back to the user that the session was not really closed, mere logging is not sufficient. This will have a cascading impact to the closeSession API exposed to the user.
>
> Deepak Barr wrote:
> Closing a session will close the lens session but not the underlying hive session. I dont think it should be ok to tell the user that the session is closed because session can not be used again anyway. What do you think ?
>
> Deepak Barr wrote:
> correction - I think it should be ok to tell the user that the session is closed because session can not be used again anyway. What do you think ?
seems fine.
- Amareshwari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review106370
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Nov. 13, 2015, 6:57 a.m., Pranav Agarwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1258
> > <https://reviews.apache.org/r/39911/diff/2/?file=1119315#file1119315line1258>
> >
> > If we indeed decide to change the semantic of this method then we should minimally communicate back to the user that the session was not really closed, mere logging is not sufficient. This will have a cascading impact to the closeSession API exposed to the user.
>
> Deepak Barr wrote:
> Closing a session will close the lens session but not the underlying hive session. I dont think it should be ok to tell the user that the session is closed because session can not be used again anyway. What do you think ?
correction - I think it should be ok to tell the user that the session is closed because session can not be used again anyway. What do you think ?
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review106370
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Puneet Gupta <pu...@gmail.com>.
> On Nov. 13, 2015, 6:57 a.m., Pranav Agarwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1245
> > <https://reviews.apache.org/r/39911/diff/2/?file=1119315#file1119315line1245>
> >
> > In my opinion we shouldn't change the default behavior of this API. Since the user asked to close the session this semantic implies that the underying queries should be killed.
> >
> > If we want to address the usecase of closing the session that are inactive and are not running any queries then driver should have an API to return the running queries for a session. And the caller should make a call whether they want to first check if there are any running queries in the driver before calling closeSession.
I was wondering when user has issued an "asynchronous" query, why should the user wait till the query finishes (sometimes more than a day). He can close the cli/session and come back later and check the status or kill the query or do other operations available via query handle. Keeping the session open when a user doesn't intend to do any further oprations other than just waiting for the async query result, can be wastage of resources for lens server. When a sync query is issued, the behaviour can be different, if really required.
- Puneet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review106370
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
> On Nov. 13, 2015, 6:57 a.m., Pranav Agarwal wrote:
> > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, line 1258
> > <https://reviews.apache.org/r/39911/diff/2/?file=1119315#file1119315line1258>
> >
> > If we indeed decide to change the semantic of this method then we should minimally communicate back to the user that the session was not really closed, mere logging is not sufficient. This will have a cascading impact to the closeSession API exposed to the user.
Closing a session will close the lens session but not the underlying hive session. I dont think it should be ok to tell the user that the session is closed because session can not be used again anyway. What do you think ?
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review106370
-----------------------------------------------------------
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Pranav Agarwal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review106370
-----------------------------------------------------------
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 114)
<https://reviews.apache.org/r/39911/#comment165122>
hiveHandle variable is tracking one-to-one mapping for QueryHandle with OperationHandle. Tracking QueryHandle is not sufficient?
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 1245)
<https://reviews.apache.org/r/39911/#comment165112>
In my opinion we shouldn't change the default behavior of this API. Since the user asked to close the session this semantic implies that the underying queries should be killed.
If we want to address the usecase of closing the session that are inactive and are not running any queries then driver should have an API to return the running queries for a session. And the caller should make a call whether they want to first check if there are any running queries in the driver before calling closeSession.
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java (line 1258)
<https://reviews.apache.org/r/39911/#comment165114>
If we indeed decide to change the semantic of this method then we should minimally communicate back to the user that the session was not really closed, mere logging is not sufficient. This will have a cascading impact to the closeSession API exposed to the user.
- Pranav Agarwal
On Nov. 8, 2015, 6:58 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2015, 6:58 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
> lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Deepak Barr <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/
-----------------------------------------------------------
(Updated Nov. 8, 2015, 6:58 p.m.)
Review request for lens.
Changes
-------
1. Added unit test.
2. Improved logging.
3. Other suggestions
Repository: lens
Description
-------
OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
Diffs (updated)
-----
lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
lens-server/src/test/java/org/apache/lens/server/query/TestQueryService.java c37b0ed
Diff: https://reviews.apache.org/r/39911/diff/
Testing
-------
Yes.
1. Created session.
2. Fired a query.
3. Deleted the session before query finishes.
4. Query keeps running and finishes eventually.
Thanks,
Deepak Barr
Re: Review Request 39911: LENS-760 : Session close should not result
in running query failures
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39911/#review105035
-----------------------------------------------------------
Can you add unit test?
- Amareshwari Sriramadasu
On Nov. 3, 2015, 6:43 p.m., Deepak Barr wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39911/
> -----------------------------------------------------------
>
> (Updated Nov. 3, 2015, 6:43 p.m.)
>
>
> Review request for lens.
>
>
> Repository: lens
>
>
> Description
> -------
>
> OperationHandle-HiveSession mapping is maintained for all active operations. Now, hive session will only be closed when there are no active operations running on that session.
>
>
> Diffs
> -----
>
> lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java c96ef20
>
> Diff: https://reviews.apache.org/r/39911/diff/
>
>
> Testing
> -------
>
> Yes.
>
> 1. Created session.
> 2. Fired a query.
> 3. Deleted the session before query finishes.
> 4. Query keeps running and finishes eventually.
>
>
> Thanks,
>
> Deepak Barr
>
>