You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@livy.apache.org by GitBox <gi...@apache.org> on 2020/04/03 22:21:04 UTC
[GitHub] [incubator-livy] tprelle-ubi opened a new pull request #289: [WIP]
[LIVY-756] Add Support Spark 3
tprelle-ubi opened a new pull request #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289
## What changes were proposed in this pull request?
Add spark 3.0 support to livy with scala 2.12 and Python 3
Fix also https://issues.apache.org/jira/browse/LIVY-423
## How was this patch tested?
Scala and Python 3 test work
I'm not able yet to make python 2 and sparkR work for spark 3 yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io commented on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.60%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.79% +2.60%
- Complexity 964 979 +15
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4231 +172
+ Misses 1314 1279 -35
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...77387dd](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.60%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.79% +2.60%
- Complexity 964 979 +15
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4231 +172
+ Misses 1314 1279 -35
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...77387dd](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.58%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.78% +2.58%
- Complexity 964 980 +16
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4230 +171
+ Misses 1314 1280 -34
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...3804692](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.60%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.79% +2.60%
- Complexity 964 979 +15
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4231 +172
+ Misses 1314 1279 -35
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...77387dd](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.60%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.79% +2.60%
- Complexity 964 979 +15
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4231 +172
+ Misses 1314 1279 -35
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...77387dd](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r434725675
##########
File path: api/src/main/java/org/apache/livy/JobContext.java
##########
@@ -41,12 +40,12 @@
/**
* @return The shared SQLContext instance.
*/
- SQLContext sqlctx();
+ SQLContext sqlctx() throws Exception ;
/**
- * @return The shared HiveContext instance.
+ * @return The shared hive SQLContext instance.
*/
- HiveContext hivectx();
Review comment:
Hi, @andrasbeni thanks for the review, I do the code in december with spark 3.0.0-preview2, I just check on spark and they push back they class in https://issues.apache.org/jira/browse/SPARK-31088, I will make the change back when they will released spark 3.0.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.58%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.78% +2.58%
- Complexity 964 980 +16
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4230 +171
+ Misses 1314 1280 -34
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...3804692](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.58%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.78% +2.58%
- Complexity 964 980 +16
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4230 +171
+ Misses 1314 1280 -34
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...3804692](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.58%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.78% +2.58%
- Complexity 964 980 +16
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4230 +171
+ Misses 1314 1280 -34
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...3804692](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.58%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.78% +2.58%
- Complexity 964 980 +16
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4230 +171
+ Misses 1314 1280 -34
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...3804692](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io edited a comment on issue #289: [WIP]
[LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #289: [WIP] [LIVY-756] Add Support Spark 3
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-608820527
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=h1) Report
> Merging [#289](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/ee7fdfc45d90c0478dcd446bc8a19a217eebe04d&el=desc) will **increase** coverage by `2.58%`.
> The diff coverage is `52.00%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #289 +/- ##
============================================
+ Coverage 68.19% 70.78% +2.58%
- Complexity 964 980 +16
============================================
Files 104 103 -1
Lines 5952 5976 +24
Branches 900 903 +3
============================================
+ Hits 4059 4230 +171
+ Misses 1314 1280 -34
+ Partials 579 466 -113
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...scala/org/apache/livy/repl/SparkRInterpreter.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TcGFya1JJbnRlcnByZXRlci5zY2FsYQ==) | `61.49% <ø> (+1.37%)` | `15.00 <0.00> (+1.00)` | |
| [...main/java/org/apache/livy/rsc/ContextLauncher.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9Db250ZXh0TGF1bmNoZXIuamF2YQ==) | `65.71% <0.00%> (-0.64%)` | `13.00 <0.00> (ø)` | |
| [...ava/org/apache/livy/rsc/driver/JobContextImpl.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvSm9iQ29udGV4dEltcGwuamF2YQ==) | `86.66% <ø> (ø)` | `16.00 <0.00> (ø)` | |
| [...java/org/apache/livy/rsc/rpc/KryoMessageCodec.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9ycGMvS3J5b01lc3NhZ2VDb2RlYy5qYXZh) | `94.54% <ø> (ø)` | `18.00 <0.00> (ø)` | |
| [...ala/org/apache/livy/scalaapi/ScalaJobContext.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkNvbnRleHQuc2NhbGE=) | `23.07% <0.00%> (ø)` | `2.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `65.11% <0.00%> (+12.17%)` | `7.00 <0.00> (ø)` | |
| [.../scala/org/apache/livy/server/SessionServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvU2Vzc2lvblNlcnZsZXQuc2NhbGE=) | `82.10% <ø> (+10.52%)` | `21.00 <0.00> (+1.00)` | |
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `72.17% <22.22%> (+1.99%)` | `51.00 <0.00> (ø)` | |
| [...java/org/apache/livy/client/common/Serializer.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-Y2xpZW50LWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvbGl2eS9jbGllbnQvY29tbW9uL1NlcmlhbGl6ZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | `3.00 <0.00> (ø)` | |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.96% <100.00%> (+0.11%)` | `8.00 <0.00> (ø)` | |
| ... and [56 more](https://codecov.io/gh/apache/incubator-livy/pull/289/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=footer). Last update [ee7fdfc...3804692](https://codecov.io/gh/apache/incubator-livy/pull/289?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] Minutis commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
Minutis commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-646481118
Spark 3 was released. https://spark.apache.org/downloads.html, please, continue :+1:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444717000
##########
File path: pom.xml
##########
@@ -86,24 +87,25 @@
<commons-codec.version>1.9</commons-codec.version>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
- <jackson.version>2.9.9</jackson.version>
+ <jackson.version>2.10.1</jackson.version>
<javax.servlet-api.version>3.1.0</javax.servlet-api.version>
<jetty.version>9.3.24.v20180605</jetty.version>
<json4s.version>3.2.11</json4s.version>
<junit.version>4.11</junit.version>
<libthrift.version>0.9.3</libthrift.version>
- <kryo.version>2.22</kryo.version>
+ <kryo.version>4.0.2</kryo.version>
<metrics.version>3.1.0</metrics.version>
- <mockito.version>1.9.5</mockito.version>
+ <mockito.version>1.10.19</mockito.version>
<netty.spark-2.11.version>4.0.37.Final</netty.spark-2.11.version>
<netty.version>${netty.spark-2.11.version}</netty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<py4j.version>0.10.7</py4j.version>
<scala-2.11.version>2.11.12</scala-2.11.version>
+ <scala-2.12.version>2.12.10</scala-2.12.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>${scala-2.11.version}</scala.version>
- <scalatest.version>2.2.4</scalatest.version>
- <scalatra.version>2.3.0</scalatra.version>
+ <scalatest.version>3.0.8</scalatest.version>
+ <scalatra.version>2.6.5</scalatra.version>
Review comment:
Scalatra added scala 2.12 support in version 2.5.0 and scalatest in 3.0.0. So the version currently in use are incompatible with Spark 3.0's Scala 2.12.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] zjffdu commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r442677351
##########
File path: assembly/pom.xml
##########
@@ -29,7 +29,7 @@
<packaging>pom</packaging>
<properties>
- <assembly.name>apache-livy-${project.version}-bin</assembly.name>
+ <assembly.name>apache-livy-${project.version}-bin-${scala.binary.version}</assembly.name>
Review comment:
to `_${scala.binary.version}` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-647246039
@tprelle is this PR ready for review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r446820287
##########
File path: scala-api/src/main/scala/org/apache/livy/scalaapi/ScalaJobHandle.scala
##########
@@ -190,6 +190,19 @@ class ScalaJobHandle[T] private[livy] (jobHandle: JobHandle[T]) extends Future[T
getJavaFutureResult(jobHandle, atMost)
this
}
+
+ // These two methods must be implemented in Scala 2.12. They're implemented as a no-op here
Review comment:
Yes, the comments seems confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] mmigdiso commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
mmigdiso commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-650323339
Hey @tprelle , will @jerryshao comments be a blocker for that PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-664907671
There still have some minor issues in JDBC side, we prepare a release when all the issues are done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648101189
I'm going to try this PR locally to see if the current code is complete or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-673399619
Hi @hongweijia, never test it with Python 3.6 but it's working well with 3.7,
Regards
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648289289
All unit and intregration scala and Python test are working.
I'm currently redeploy it to make more integration test because of https://issues.apache.org/jira/browse/SPARK-29748 (the biggest change I spot for livy between 3.0.0-preview2 version en 3.0.0 version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] hongweijia commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
hongweijia commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-673302574
Hi @tprelle
Has livy already supported Spark 3 with Python 3.6?
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] stczwd commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
stczwd commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-657356354
Glad to see this patch merged. I have test this path, it works well with spark3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444876495
##########
File path: api/src/main/java/org/apache/livy/JobContext.java
##########
@@ -41,12 +41,12 @@
/**
* @return The shared SQLContext instance.
*/
- SQLContext sqlctx();
+ SQLContext sqlctx() throws Exception ;
Review comment:
It was before https://issues.apache.org/jira/browse/SPARK-31088, when they put back HiveContext class back to spark.
When I revert the change I forgot to revert also the method signature.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444880828
##########
File path: client-http/pom.xml
##########
@@ -109,58 +109,6 @@
<build>
<plugins>
- <plugin>
Review comment:
It was a mistake, i will put it back
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r446593365
##########
File path: scala-api/src/main/scala/org/apache/livy/scalaapi/ScalaJobHandle.scala
##########
@@ -190,6 +190,19 @@ class ScalaJobHandle[T] private[livy] (jobHandle: JobHandle[T]) extends Future[T
getJavaFutureResult(jobHandle, atMost)
this
}
+
+ // These two methods must be implemented in Scala 2.12. They're implemented as a no-op here
Review comment:
Would you please elaborate why the below two methods are needed for Scala 2.12?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648547180
I just did a quick review about the overall implementation. Seems like we can only support one Scala version for one build here in this PR. But Livy could actually support different versions of Scala in one build (we used to support Scala 2.10 and 2.11 for one build). So I think we should have both Scala 2.12 and 2.11 support, and let Livy to choose jars automatically based on the Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] giftkugel commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
giftkugel commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-664894916
Is there any timeline to add this patch into a release? Maybe Livy 0.8?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427
##########
File path: pom.xml
##########
@@ -86,24 +87,25 @@
<commons-codec.version>1.9</commons-codec.version>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
- <jackson.version>2.9.9</jackson.version>
+ <jackson.version>2.10.1</jackson.version>
<javax.servlet-api.version>3.1.0</javax.servlet-api.version>
<jetty.version>9.3.24.v20180605</jetty.version>
<json4s.version>3.2.11</json4s.version>
<junit.version>4.11</junit.version>
<libthrift.version>0.9.3</libthrift.version>
- <kryo.version>2.22</kryo.version>
+ <kryo.version>4.0.2</kryo.version>
<metrics.version>3.1.0</metrics.version>
- <mockito.version>1.9.5</mockito.version>
+ <mockito.version>1.10.19</mockito.version>
<netty.spark-2.11.version>4.0.37.Final</netty.spark-2.11.version>
<netty.version>${netty.spark-2.11.version}</netty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<py4j.version>0.10.7</py4j.version>
<scala-2.11.version>2.11.12</scala-2.11.version>
+ <scala-2.12.version>2.12.10</scala-2.12.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>${scala-2.11.version}</scala.version>
- <scalatest.version>2.2.4</scalatest.version>
- <scalatra.version>2.3.0</scalatra.version>
+ <scalatest.version>3.0.8</scalatest.version>
+ <scalatra.version>2.6.5</scalatra.version>
Review comment:
Scalatra is only used by Livy Server, we don't have to upgrade Livy Server's Scala version to 2.12, we could leave the Livy Server Scala upgrade to another JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648102517
@jerryshao the code it's working for spark scala and python but I do not managed to make it work for sparkR.
I can submit the PR and open a ticket for sparkR support for spark3 if it's ok for the community
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648813898
> I just did a quick review about the overall implementation. Seems like we can only support one Scala version for one build here in this PR. But Livy could actually support different versions of Scala in one build (we used to support Scala 2.10 and 2.11 for one build). So I think we should have both Scala 2.12 and 2.11 support, and let Livy to choose jars automatically based on the Spark.
I like the idea, I try it, but I was not able to make it work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648103728
Yes, I think it is fine to leave R support to another PR. Please make sure this PR is OK for Scala and Python part.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-648718274
I'm doing some local improvements based on @tprelle 's current PR, I can submit a PR when everything is ready.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444770427
##########
File path: pom.xml
##########
@@ -86,24 +87,25 @@
<commons-codec.version>1.9</commons-codec.version>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
- <jackson.version>2.9.9</jackson.version>
+ <jackson.version>2.10.1</jackson.version>
<javax.servlet-api.version>3.1.0</javax.servlet-api.version>
<jetty.version>9.3.24.v20180605</jetty.version>
<json4s.version>3.2.11</json4s.version>
<junit.version>4.11</junit.version>
<libthrift.version>0.9.3</libthrift.version>
- <kryo.version>2.22</kryo.version>
+ <kryo.version>4.0.2</kryo.version>
<metrics.version>3.1.0</metrics.version>
- <mockito.version>1.9.5</mockito.version>
+ <mockito.version>1.10.19</mockito.version>
<netty.spark-2.11.version>4.0.37.Final</netty.spark-2.11.version>
<netty.version>${netty.spark-2.11.version}</netty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<py4j.version>0.10.7</py4j.version>
<scala-2.11.version>2.11.12</scala-2.11.version>
+ <scala-2.12.version>2.12.10</scala-2.12.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>${scala-2.11.version}</scala.version>
- <scalatest.version>2.2.4</scalatest.version>
- <scalatra.version>2.3.0</scalatra.version>
+ <scalatest.version>3.0.8</scalatest.version>
+ <scalatra.version>2.6.5</scalatra.version>
Review comment:
Scalatra is only used by Livy Server, we don't have to upgrade Livy Server's Scala version to 2.12, we could leave the Livy Server Scala support to another JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] zjffdu commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
zjffdu commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r442677351
##########
File path: assembly/pom.xml
##########
@@ -29,7 +29,7 @@
<packaging>pom</packaging>
<properties>
- <assembly.name>apache-livy-${project.version}-bin</assembly.name>
+ <assembly.name>apache-livy-${project.version}-bin-${scala.binary.version}</assembly.name>
Review comment:
Use `_${scala.binary.version}` to align with convention
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444774680
##########
File path: api/src/main/java/org/apache/livy/JobContext.java
##########
@@ -41,12 +41,12 @@
/**
* @return The shared SQLContext instance.
*/
- SQLContext sqlctx();
+ SQLContext sqlctx() throws Exception ;
Review comment:
Yes, I have the same concern about changing the signature of this interface.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444719602
##########
File path: api/src/main/java/org/apache/livy/JobContext.java
##########
@@ -41,12 +41,12 @@
/**
* @return The shared SQLContext instance.
*/
- SQLContext sqlctx();
+ SQLContext sqlctx() throws Exception ;
Review comment:
I believe adding throws clauses to these methods could break Java client applications. I suggest to wrap exceptions that are not RuntimeException in a RuntimeException instead.
##########
File path: pom.xml
##########
@@ -86,24 +87,25 @@
<commons-codec.version>1.9</commons-codec.version>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
- <jackson.version>2.9.9</jackson.version>
+ <jackson.version>2.10.1</jackson.version>
<javax.servlet-api.version>3.1.0</javax.servlet-api.version>
<jetty.version>9.3.24.v20180605</jetty.version>
<json4s.version>3.2.11</json4s.version>
<junit.version>4.11</junit.version>
<libthrift.version>0.9.3</libthrift.version>
- <kryo.version>2.22</kryo.version>
+ <kryo.version>4.0.2</kryo.version>
<metrics.version>3.1.0</metrics.version>
- <mockito.version>1.9.5</mockito.version>
+ <mockito.version>1.10.19</mockito.version>
<netty.spark-2.11.version>4.0.37.Final</netty.spark-2.11.version>
<netty.version>${netty.spark-2.11.version}</netty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<py4j.version>0.10.7</py4j.version>
<scala-2.11.version>2.11.12</scala-2.11.version>
+ <scala-2.12.version>2.12.10</scala-2.12.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>${scala-2.11.version}</scala.version>
- <scalatest.version>2.2.4</scalatest.version>
- <scalatra.version>2.3.0</scalatra.version>
+ <scalatest.version>3.0.8</scalatest.version>
+ <scalatra.version>2.6.5</scalatra.version>
Review comment:
Scalatra added scala 2.12 support in version 2.5.0 and scalatest in 3.0.0. So the version currently in use are incompatibel with Spark 3.0's Scala 2.12.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444614548
##########
File path: .travis.yml
##########
@@ -34,6 +34,12 @@ matrix:
env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipITs'
- name: "Spark 2.4 ITs"
env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipTests'
+ - name: "Spark 3.0 Unit Tests"
+ env: MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipITs'
+ - name: "Spark 3.0 ITs"
+ env:
+ - MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipTests'
+ - PYSPARK_ROW_FIELD_SORTING_ENABLED=true
Review comment:
What is the purpose of setting this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444628517
##########
File path: pom.xml
##########
@@ -86,24 +87,25 @@
<commons-codec.version>1.9</commons-codec.version>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
- <jackson.version>2.9.9</jackson.version>
+ <jackson.version>2.10.1</jackson.version>
<javax.servlet-api.version>3.1.0</javax.servlet-api.version>
<jetty.version>9.3.24.v20180605</jetty.version>
<json4s.version>3.2.11</json4s.version>
<junit.version>4.11</junit.version>
<libthrift.version>0.9.3</libthrift.version>
- <kryo.version>2.22</kryo.version>
+ <kryo.version>4.0.2</kryo.version>
<metrics.version>3.1.0</metrics.version>
- <mockito.version>1.9.5</mockito.version>
+ <mockito.version>1.10.19</mockito.version>
<netty.spark-2.11.version>4.0.37.Final</netty.spark-2.11.version>
<netty.version>${netty.spark-2.11.version}</netty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<py4j.version>0.10.7</py4j.version>
<scala-2.11.version>2.11.12</scala-2.11.version>
+ <scala-2.12.version>2.12.10</scala-2.12.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>${scala-2.11.version}</scala.version>
- <scalatest.version>2.2.4</scalatest.version>
- <scalatra.version>2.3.0</scalatra.version>
+ <scalatest.version>3.0.8</scalatest.version>
+ <scalatra.version>2.6.5</scalatra.version>
Review comment:
Are this artifacts upgrading necessary for Spark 3.0 support?
##########
File path: pom.xml
##########
@@ -86,24 +87,25 @@
<commons-codec.version>1.9</commons-codec.version>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
- <jackson.version>2.9.9</jackson.version>
+ <jackson.version>2.10.1</jackson.version>
<javax.servlet-api.version>3.1.0</javax.servlet-api.version>
<jetty.version>9.3.24.v20180605</jetty.version>
<json4s.version>3.2.11</json4s.version>
<junit.version>4.11</junit.version>
<libthrift.version>0.9.3</libthrift.version>
- <kryo.version>2.22</kryo.version>
+ <kryo.version>4.0.2</kryo.version>
<metrics.version>3.1.0</metrics.version>
- <mockito.version>1.9.5</mockito.version>
+ <mockito.version>1.10.19</mockito.version>
<netty.spark-2.11.version>4.0.37.Final</netty.spark-2.11.version>
<netty.version>${netty.spark-2.11.version}</netty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<py4j.version>0.10.7</py4j.version>
<scala-2.11.version>2.11.12</scala-2.11.version>
+ <scala-2.12.version>2.12.10</scala-2.12.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>${scala-2.11.version}</scala.version>
- <scalatest.version>2.2.4</scalatest.version>
- <scalatra.version>2.3.0</scalatra.version>
+ <scalatest.version>3.0.8</scalatest.version>
+ <scalatra.version>2.6.5</scalatra.version>
Review comment:
Are these artifacts upgrading necessary for Spark 3.0 support?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-646867581
> Spark 3 was released. https://spark.apache.org/downloads.html, please, continue 👍
Move to spark 3.0.0.
Do not have time yet to test spark R
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao edited a comment on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao edited a comment on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-664907671
There still have some minor issues in JDBC side, will prepare a release when all the issues are done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444618359
##########
File path: .travis.yml
##########
@@ -34,6 +34,12 @@ matrix:
env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipITs'
- name: "Spark 2.4 ITs"
env: MVN_FLAG='-Pspark-2.4 -Pthriftserver -DskipTests'
+ - name: "Spark 3.0 Unit Tests"
+ env: MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipITs'
+ - name: "Spark 3.0 ITs"
+ env:
+ - MVN_FLAG='-Pthriftserver -Pspark-3.0 -DskipTests'
+ - PYSPARK_ROW_FIELD_SORTING_ENABLED=true
Review comment:
Got it, seems an incompatible change in Spark 3.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao closed pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao closed pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444613917
##########
File path: client-http/pom.xml
##########
@@ -109,58 +109,6 @@
<build>
<plugins>
- <plugin>
Review comment:
What's the reason to remove this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r446670999
##########
File path: scala-api/src/main/scala/org/apache/livy/scalaapi/ScalaJobHandle.scala
##########
@@ -190,6 +190,19 @@ class ScalaJobHandle[T] private[livy] (jobHandle: JobHandle[T]) extends Future[T
getJavaFutureResult(jobHandle, atMost)
this
}
+
+ // These two methods must be implemented in Scala 2.12. They're implemented as a no-op here
Review comment:
In Scala 2.12 these two new methods ([see source](https://github.com/scala/scala/blob/v2.12.10/src/library/scala/concurrent/Future.scala#L248-L268 )) were added to the trait `Future`. As a result, `ScalaJobHandle`, which extends Future needs to implement them.
At the same time I am not sure what @tprelle means by "the two subclasses below". AFAICT `ScalaJobHandle` does not have any subclasses.
Also, I wouldn't say throwing an exception is a no-op. I suggest "They are implemented as unsupported operations here" if they are really not going to be [implemented](https://github.com/scala/scala/blob/v2.12.10/src/library/scala/concurrent/impl/Promise.scala#L31-L48).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] jerryshao commented on pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
jerryshao commented on pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#issuecomment-652848316
Thanks @tprelle for your work, I've already merged this PR, all credits to you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] andrasbeni commented on a change in pull request #289: [WIP] [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
andrasbeni commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r434684045
##########
File path: api/src/main/java/org/apache/livy/JobContext.java
##########
@@ -41,12 +40,12 @@
/**
* @return The shared SQLContext instance.
*/
- SQLContext sqlctx();
+ SQLContext sqlctx() throws Exception ;
/**
- * @return The shared HiveContext instance.
+ * @return The shared hive SQLContext instance.
*/
- HiveContext hivectx();
Review comment:
This will break compatibility to those who keep using Livy with Spark 2.4.
I'd recommend to maintain two versions of this file and adding the correct version to compile sources based on profiles.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-livy] tprelle commented on a change in pull request #289: [LIVY-756] Add Support Spark 3
Posted by GitBox <gi...@apache.org>.
tprelle commented on a change in pull request #289:
URL: https://github.com/apache/incubator-livy/pull/289#discussion_r444891736
##########
File path: pom.xml
##########
@@ -86,24 +87,25 @@
<commons-codec.version>1.9</commons-codec.version>
<httpclient.version>4.5.3</httpclient.version>
<httpcore.version>4.4.4</httpcore.version>
- <jackson.version>2.9.9</jackson.version>
+ <jackson.version>2.10.1</jackson.version>
<javax.servlet-api.version>3.1.0</javax.servlet-api.version>
<jetty.version>9.3.24.v20180605</jetty.version>
<json4s.version>3.2.11</json4s.version>
<junit.version>4.11</junit.version>
<libthrift.version>0.9.3</libthrift.version>
- <kryo.version>2.22</kryo.version>
+ <kryo.version>4.0.2</kryo.version>
<metrics.version>3.1.0</metrics.version>
- <mockito.version>1.9.5</mockito.version>
+ <mockito.version>1.10.19</mockito.version>
<netty.spark-2.11.version>4.0.37.Final</netty.spark-2.11.version>
<netty.version>${netty.spark-2.11.version}</netty.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<py4j.version>0.10.7</py4j.version>
<scala-2.11.version>2.11.12</scala-2.11.version>
+ <scala-2.12.version>2.12.10</scala-2.12.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>${scala-2.11.version}</scala.version>
- <scalatest.version>2.2.4</scalatest.version>
- <scalatra.version>2.3.0</scalatra.version>
+ <scalatest.version>3.0.8</scalatest.version>
+ <scalatra.version>2.6.5</scalatra.version>
Review comment:
I try to limit the number of version change, but because of scala 2.12 like @andrasbeni says I need to use scalatest 3.0.8 who are some breaking change like org.scalatest.Suite =>org.scalatest.TestSuite, so to do not copy the class I need to change also the global version of scalatest. So I need to change the version of scalatra because of the incompatibility between scalatest 3.0.8 and scalatra-scalatest 2.3.0.
As I was working and easier to move forward with these upgrade of version I choose to submit like this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org