You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@livy.apache.org by GitBox <gi...@apache.org> on 2020/02/16 05:11:02 UTC
[GitHub] [incubator-livy] shanyu opened a new pull request #280: [LIVY-749]
Remove explicitly adding datanucleus jars
shanyu opened a new pull request #280: [LIVY-749] Remove explicitly adding datanucleus jars
URL: https://github.com/apache/incubator-livy/pull/280
## What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/LIVY-749
If we start any Livy session with hive support (livy.repl.enable-hive-context=true), we see that 3 datanucleus jars are uploaded to HDFS and downloaded to drivers/executors:
Uploading resource file:/opt/spark/jars/datanucleus-api-jdo-3.2.6.jar -> hdfs://namenode/user/test1/.sparkStaging/application_1581024490249_0002/datanucleus-api-jdo-3.2.6.jar
...
These 3 datanucleus jars are not needed bacause they are already included in Spark 2.x jars folder.
The reason is because in InteractiveSession.scala, method mergeHiveSiteAndHiveDeps(), we merged datanucleus jars to spark.jars list with method datanucleusJars(). We should remove datanucleusJars() function.
## How was this patch tested?
existing unit/integration tests, and manual tests.
Please review https://livy.incubator.apache.org/community/ before opening a pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-io commented on issue #280: [LIVY-749]
Remove explicitly adding datanucleus jars
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #280: [LIVY-749] Remove explicitly adding datanucleus jars
URL: https://github.com/apache/incubator-livy/pull/280#issuecomment-586672062
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=h1) Report
> Merging [#280](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/d6d8261c89aa56f09ddc7ef962360d106c047063?src=pr&el=desc) will **decrease** coverage by `0.13%`.
> The diff coverage is `59.86%`.
[](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #280 +/- ##
===========================================
- Coverage 68.54% 68.4% -0.14%
- Complexity 906 964 +58
===========================================
Files 100 104 +4
Lines 5674 5934 +260
Branches 854 895 +41
===========================================
+ Hits 3889 4059 +170
- Misses 1228 1295 +67
- Partials 557 580 +23
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...ain/java/org/apache/livy/rsc/driver/Statement.java](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvU3RhdGVtZW50LmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
| [...in/scala/org/apache/livy/server/ui/UIServlet.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvdWkvVUlTZXJ2bGV0LnNjYWxh) | `4.49% <0%> (ø)` | `1 <0> (ø)` | :arrow_down: |
| [...a/org/apache/livy/utils/LivyUncaughtException.java](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9MaXZ5VW5jYXVnaHRFeGNlcHRpb24uamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | |
| [...rver/src/main/scala/org/apache/livy/LivyConf.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9MaXZ5Q29uZi5zY2FsYQ==) | `96.13% <100%> (+0.28%)` | `21 <0> (ø)` | :arrow_down: |
| [...che/livy/server/recovery/ZooKeeperStateStore.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvcmVjb3ZlcnkvWm9vS2VlcGVyU3RhdGVTdG9yZS5zY2FsYQ==) | `100% <100%> (+20.93%)` | `8 <8> (-9)` | :arrow_down: |
| [rsc/src/main/java/org/apache/livy/rsc/Utils.java](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9VdGlscy5qYXZh) | `81.57% <100%> (-1.35%)` | `14 <1> (-1)` | |
| [.../src/main/scala/org/apache/livy/repl/Session.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-cmVwbC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2xpdnkvcmVwbC9TZXNzaW9uLnNjYWxh) | `67.69% <100%> (+0.5%)` | `37 <0> (ø)` | :arrow_down: |
| [rsc/src/main/java/org/apache/livy/rsc/RSCConf.java](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDb25mLmphdmE=) | `87.85% <100%> (+2%)` | `8 <0> (+1)` | :arrow_up: |
| [...main/scala/org/apache/livy/server/LivyServer.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvTGl2eVNlcnZlci5zY2FsYQ==) | `33.03% <13.63%> (-2.44%)` | `11 <0> (ø)` | |
| [...c/src/main/java/org/apache/livy/rsc/RSCClient.java](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9SU0NDbGllbnQuamF2YQ==) | `73.49% <16.66%> (-1.67%)` | `20 <1> (ø)` | |
| ... and [32 more](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=footer). Last update [d6d8261...d147fc9](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-livy] codecov-commenter commented on pull request #280: [LIVY-749] Remove explicitly adding datanucleus jars
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #280:
URL: https://github.com/apache/incubator-livy/pull/280#issuecomment-632516454
# [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=h1) Report
> Merging [#280](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-livy/commit/3a2685643670c1cd1530feb7a54516b01784e781&el=desc) will **increase** coverage by `0.19%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #280 +/- ##
============================================
+ Coverage 68.20% 68.40% +0.19%
+ Complexity 965 964 -1
============================================
Files 104 104
Lines 5951 5934 -17
Branches 900 895 -5
============================================
Hits 4059 4059
+ Misses 1314 1295 -19
- Partials 578 580 +2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...e/livy/server/interactive/InteractiveSession.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zZXJ2ZXIvaW50ZXJhY3RpdmUvSW50ZXJhY3RpdmVTZXNzaW9uLnNjYWxh) | `73.84% <ø> (+3.67%)` | `51.00 <0.00> (ø)` | |
| [...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2NhbGEtYXBpL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS9zY2FsYWFwaS9TY2FsYUpvYkhhbmRsZS5zY2FsYQ==) | `52.94% <0.00%> (-2.95%)` | `7.00% <0.00%> (ø%)` | |
| [...ain/java/org/apache/livy/rsc/driver/RSCDriver.java](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-cnNjL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saXZ5L3JzYy9kcml2ZXIvUlNDRHJpdmVyLmphdmE=) | `80.00% <0.00%> (-0.84%)` | `45.00% <0.00%> (-1.00%)` | |
| [...ain/scala/org/apache/livy/utils/SparkYarnApp.scala](https://codecov.io/gh/apache/incubator-livy/pull/280/diff?src=pr&el=tree#diff-c2VydmVyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGl2eS91dGlscy9TcGFya1lhcm5BcHAuc2NhbGE=) | `73.75% <0.00%> (+1.87%)` | `40.00% <0.00%> (ø%)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=footer). Last update [3a26856...d147fc9](https://codecov.io/gh/apache/incubator-livy/pull/280?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org