You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@livy.apache.org by GitBox <gi...@apache.org> on 2020/01/20 08:08:42 UTC
[GitHub] [incubator-livy] yiheng opened a new pull request #277: [LIVY-724]
Support Session Lazy Recover
yiheng opened a new pull request #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277
## What changes were proposed in this pull request?
In this patch, we support a new session recovery mode. In this mode, the sessions on the failed server will not be recovered until a request for it arrives. This mode will be enabled in the multi-active HA scenario.
## How was this patch tested?
New and existing unit tests.
Please review https://livy.incubator.apache.org/community/ before opening a pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] yiheng commented on issue #277: [LIVY-724] Support
Session Lazy Recover
Posted by GitBox <gi...@apache.org>.
yiheng commented on issue #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277#issuecomment-576155790
@runzhiwang and @jerryshao Please help review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] yiheng commented on issue #277: [LIVY-724] Support
Session Lazy Recover
Posted by GitBox <gi...@apache.org>.
yiheng commented on issue #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277#issuecomment-605750168
ping @jerryshao .Could you help reivew it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] yiheng closed pull request #277: [LIVY-724]
Support Session Lazy Recover
Posted by GitBox <gi...@apache.org>.
yiheng closed pull request #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #277:
[LIVY-724] Support Session Lazy Recover
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277#issuecomment-598531542
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=h1) Report
> Merging [#277](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/66b5833e413bc10e39e3b92b585f496444c147d4&el=desc) will **increase** coverage by `0.50%`.
> The diff coverage is `70.58%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #277 +/- ##
============================================
+ Coverage 68.14% 68.65% +0.50%
- Complexity 959 983 +24
============================================
Files 104 104
Lines 5946 6032 +86
Branches 899 922 +23
============================================
+ Hits 4052 4141 +89
+ Misses 1312 1305 -7
- Partials 582 586 +4
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `80.00% <50.00%> (-1.82%)` | `31.00 <4.00> (+4.00)` | :arrow_down: |
| [...org/apache/livy/server/recovery/SessionStore.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvcmVjb3ZlcnkvU2Vzc2lvblN0b3JlLnNjYWxh) | `80.00% <80.00%> (ø)` | `11.00 <1.00> (+1.00)` | |
| [...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==) | `96.20% <100.00%> (+0.07%)` | `21.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0.00%> (-2.95%)` | `7.00% <0.00%> (ø%)` | |
| [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `33.03% <0.00%> (-0.45%)` | `11.00% <0.00%> (ø%)` | |
| [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.83% <0.00%> (+0.83%)` | `46.00% <0.00%> (+1.00%)` | |
| [...c/main/java/org/apache/livy/LivyClientBuilder.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-YXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L0xpdnlDbGllbnRCdWlsZGVyLmphdmE=) | `89.47% <0.00%> (+6.42%)` | `28.00% <0.00%> (+11.00%)` | |
| [...va/org/apache/livy/client/http/LivyConnection.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-Y2xpZW50LWh0dHAvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2xpdnkvY2xpZW50L2h0dHAvTGl2eUNvbm5lY3Rpb24uamF2YQ==) | `90.26% <0.00%> (+8.21%)` | `20.00% <0.00%> (+5.00%)` | |
| [...ain/java/org/apache/livy/rsc/RSCClientFactory.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnRGYWN0b3J5LmphdmE=) | `74.41% <0.00%> (+9.30%)` | `11.00% <0.00%> (+2.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=footer). Last update [66b5833...02c4d2a](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io commented on issue #277: [LIVY-724]
Support Session Lazy Recover
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277#issuecomment-598531542
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=h1) Report
> Merging [#277](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/66b5833e413bc10e39e3b92b585f496444c147d4&el=desc) will **increase** coverage by `0.50%`.
> The diff coverage is `70.58%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #277 +/- ##
============================================
+ Coverage 68.14% 68.65% +0.50%
- Complexity 959 983 +24
============================================
Files 104 104
Lines 5946 6032 +86
Branches 899 922 +23
============================================
+ Hits 4052 4141 +89
+ Misses 1312 1305 -7
- Partials 582 586 +4
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `80.00% <50.00%> (-1.82%)` | `31.00 <4.00> (+4.00)` | :arrow_down: |
| [...org/apache/livy/server/recovery/SessionStore.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvcmVjb3ZlcnkvU2Vzc2lvblN0b3JlLnNjYWxh) | `80.00% <80.00%> (ø)` | `11.00 <1.00> (+1.00)` | |
| [...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==) | `96.20% <100.00%> (+0.07%)` | `21.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0.00%> (-2.95%)` | `7.00% <0.00%> (ø%)` | |
| [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `33.03% <0.00%> (-0.45%)` | `11.00% <0.00%> (ø%)` | |
| [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.83% <0.00%> (+0.83%)` | `46.00% <0.00%> (+1.00%)` | |
| [...c/main/java/org/apache/livy/LivyClientBuilder.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-YXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L0xpdnlDbGllbnRCdWlsZGVyLmphdmE=) | `89.47% <0.00%> (+6.42%)` | `28.00% <0.00%> (+11.00%)` | |
| [...va/org/apache/livy/client/http/LivyConnection.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-Y2xpZW50LWh0dHAvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2xpdnkvY2xpZW50L2h0dHAvTGl2eUNvbm5lY3Rpb24uamF2YQ==) | `90.26% <0.00%> (+8.21%)` | `20.00% <0.00%> (+5.00%)` | |
| [...ain/java/org/apache/livy/rsc/RSCClientFactory.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnRGYWN0b3J5LmphdmE=) | `74.41% <0.00%> (+9.30%)` | `11.00% <0.00%> (+2.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=footer). Last update [66b5833...02c4d2a](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] yiheng opened a new pull request #277: [LIVY-724]
Support Session Lazy Recover
Posted by GitBox <gi...@apache.org>.
yiheng opened a new pull request #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277
## What changes were proposed in this pull request?
In this patch, we support a new session recovery mode. In this mode, the sessions on the failed server will not be recovered until a request for it arrives. This mode will be enabled in the multi-active HA scenario.
## How was this patch tested?
New and existing unit tests.
Please review https://livy.incubator.apache.org/community/ before opening a pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #277:
[LIVY-724] Support Session Lazy Recover
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #277: [LIVY-724] Support Session Lazy Recover
URL: https://github.com/apache/incubator-livy/pull/277#issuecomment-598531542
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=h1) Report
> Merging [#277](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/66b5833e413bc10e39e3b92b585f496444c147d4&el=desc) will **increase** coverage by `0.50%`.
> The diff coverage is `70.58%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #277 +/- ##
============================================
+ Coverage 68.14% 68.65% +0.50%
- Complexity 959 983 +24
============================================
Files 104 104
Lines 5946 6032 +86
Branches 899 922 +23
============================================
+ Hits 4052 4141 +89
+ Misses 1312 1305 -7
- Partials 582 586 +4
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...cala/org/apache/livy/sessions/SessionManager.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXNzaW9ucy9TZXNzaW9uTWFuYWdlci5zY2FsYQ==) | `80.00% <50.00%> (-1.82%)` | `31.00 <4.00> (+4.00)` | :arrow_down: |
| [...org/apache/livy/server/recovery/SessionStore.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvcmVjb3ZlcnkvU2Vzc2lvblN0b3JlLnNjYWxh) | `80.00% <80.00%> (ø)` | `11.00 <1.00> (+1.00)` | |
| [...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==) | `96.20% <100.00%> (+0.07%)` | `21.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0.00%> (-2.95%)` | `7.00% <0.00%> (ø%)` | |
| [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `33.03% <0.00%> (-0.45%)` | `11.00% <0.00%> (ø%)` | |
| [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.83% <0.00%> (+0.83%)` | `46.00% <0.00%> (+1.00%)` | |
| [...c/main/java/org/apache/livy/LivyClientBuilder.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-YXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L0xpdnlDbGllbnRCdWlsZGVyLmphdmE=) | `89.47% <0.00%> (+6.42%)` | `28.00% <0.00%> (+11.00%)` | |
| [...va/org/apache/livy/client/http/LivyConnection.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-Y2xpZW50LWh0dHAvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2xpdnkvY2xpZW50L2h0dHAvTGl2eUNvbm5lY3Rpb24uamF2YQ==) | `90.26% <0.00%> (+8.21%)` | `20.00% <0.00%> (+5.00%)` | |
| [...ain/java/org/apache/livy/rsc/RSCClientFactory.java](https://codecov.io/gh/apache/incubator-livy/pull/277/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnRGYWN0b3J5LmphdmE=) | `74.41% <0.00%> (+9.30%)` | `11.00% <0.00%> (+2.00%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=footer). Last update [66b5833...02c4d2a](https://codecov.io/gh/apache/incubator-livy/pull/277?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services