You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by calvinjia <gi...@git.apache.org> on 2015/04/03 21:06:17 UTC
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
GitHub user calvinjia opened a pull request:
https://github.com/apache/spark/pull/5354
[SPARK-6122][Core] Upgrade tachyon-client version to 0.6.3
This is a reopening of #4867.
A short summary of the issues resolved from the previous PR:
1. HTTPClient version mismatch: Selenium (used for UI tests) requires version 4.3.x, and Tachyon included 4.2.5 through a transitive dependency of its shaded thrift jar. To address this, Tachyon 0.6.3 will promote the transitive dependencies of the shaded jar so they can be excluded in spark.
2. Jackson-Mapper-ASL version mismatch: In lower versions of hadoop-client (ie. 1.0.4), version 1.0.1 is included. The parquet library used in spark sql requires version 1.8+. Its unclear to me why upgrading tachyon-client would cause this dependency to break. The solution was to exclude jackson-mapper-asl from hadoop-client.
It seems that the dependency management in spark-parent will not work on transitive dependencies, one way to make sure jackson-mapper-asl is included with the correct version is to add it as a top level dependency. The best solution would be to exclude the dependency in the modules which require a higher version, but that did not fix the unit tests. Any suggestions on the best way to solve this would be appreciated!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/calvinjia/spark upgrade_tachyon_0.6.3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5354.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5354
----
commit 71bf4413ea58cb0d8bcd14d8c6f84c4a7fdbcc6a
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-03-03T03:08:37Z
Upgrade Tachyon client version to 0.6.0.
commit 11907b34e4b70a640944164b9db66cf507f3918c
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-03-04T03:18:46Z
Use TachyonURI for tachyon paths instead of strings.
commit eed9230ab0087f3ffd7d36816644780ba1d7b909
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-03-18T20:27:51Z
Update tachyon version to 0.6.1.
commit 910fabdc5e9324752d9a429acb4b6f7fe66aa9b6
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-03-24T23:56:22Z
Update to master
commit a8a923c6382a0849c37847570189f7c88dfdf345
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-03-30T21:29:12Z
Exclude httpcomponents from Tachyon
commit a93c94fc8157c543190e2654e14ab83aeb98720b
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-03-30T23:51:11Z
Exclude jackson-mapper-asl from hadoop client since it has a lower version than spark's expected version.
commit a204df94f296e8b8eb2b6093ea8be285bd6a7ac9
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-04-01T18:16:13Z
Update make distribution tachyon version.
commit 0ae6c97f97ce0e5f3b8e51ddbdfa39f33069a5c7
Author: Calvin Jia <ji...@gmail.com>
Date: 2015-04-03T18:43:46Z
Change tachyon version to 0.6.3
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-90192443
@JoshRosen @haoyuan
Thanks for retesting, I've updated the pom to exclude the promoted dependencies of jetty and curator.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94866193
Why would the Tachyon version change itself alter the jackson or httpclient dependencies? was it already not working?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95799161
@srowen
I appreciate the feedback, and I've cleaned up the httpclient versions as you suggested.
Do you have any other comments? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94417835
I thought this was about Jackson, since that's what's excluded. It's already managed to version 1.9.13, yes. Is that not the version you need? This is a version that I think a lot of Hadoop components agree on, so would be good if that's also fine for Tachyon.
I don't see anything bringing in httpclient 4.2.5 at the moment. In fact I was confused to see `commons.httpclient.version` doesn't do anything. This actually appears to be added only for Kinesis, and only affects its dependencies.
@ScrapCodes (looks like you added that bit) this seems pretty confusing since it doesn't affect the version used in the rest of the project, which looks to be some mismatched combination of core 4.3.3 and client 4.3.6.
Since it seems Tachyon wants 4.3.2, can we all agree on `org.apache.httpcomponents:httpclient` and `org.apache.httpcomponents:httpcore` version 4.3.6?
So: we would explicitly manage both of these versions in the parent `pom.xml` and remove the two manual version management stanzas for Kinesis (the two refs to `${commons.httpclient.version}` in children).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-90249975
[Test build #29753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29753/consoleFull) for PR 5354 at commit [`e2ff80a`](https://github.com/apache/spark/commit/e2ff80af7a1793c10ab090350bb28111706464fe).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-90708752
@JoshRosen @aarondav @pwendell
What do you think about the current PR? Is there any better strategy for handling the Jackson-mapper-asl issue?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92747236
I think I'm not getting across the first point: why can't you solve this with an entry in `dependencyManagement` that declares the desired version of thrift to include, for example? That particular example might be tricky, but, then simpler examples like `commons-beanutils`. I am 95% sure this is not necessary, like much of this.
My second point is still different, and it's a question about `tachyon-client` itself. Why does it depend on these things if they're not used, and can that be fixed upstream?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95799433
[Test build #30915 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30915/consoleFull) for PR 5354 at commit [`0eefe4d`](https://github.com/apache/spark/commit/0eefe4d46c0a42859b8c9c0bc0ff98a0beeb440a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-90277261
[Test build #29753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29753/consoleFull) for PR 5354 at commit [`e2ff80a`](https://github.com/apache/spark/commit/e2ff80af7a1793c10ab090350bb28111706464fe).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `tachyon-0.6.3.jar`
* `tachyon-client-0.6.3.jar`
* This patch **removes the following dependencies:**
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89850024
[Test build #29731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29731/consoleFull) for PR 5354 at commit [`a3a29da`](https://github.com/apache/spark/commit/a3a29da67ef7926f80bc4dcb410e090e0d6f7cb9).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `jetty-continuation-8.1.14.v20131031.jar`
* `jetty-http-8.1.14.v20131031.jar`
* `jetty-io-7.6.15.v20140411.jar`
* `jetty-security-8.1.14.v20131031.jar`
* `jetty-util-8.1.14.v20131031.jar`
* `jetty-xml-7.6.15.v20140411.jar`
* `tachyon-0.6.3.jar`
* `tachyon-client-0.6.3.jar`
* This patch **removes the following dependencies:**
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95514167
I don't see a Jackson-related failure, even when directly running `org.apache.spark.mllib.regression.RidgeRegressionSuite`, with the changes in this patch.
However I do see the HttpClient problem:
```
[info] Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.UISeleniumSuite *** ABORTED *** (106 milliseconds)
[info] java.lang.NoSuchMethodError: org.apache.http.impl.cookie.BrowserCompatSpecFactory.create(Lorg/apache/http/protocol/HttpContext;)Lorg/apache/http/cookie/CookieSpec;
```
The change here works, although I think it can be tightened up a little bit. My last experiment will be to see if we can get away with `runtime` scope instead; `test` scope didn't work. I'll post what I have in a branch for you to take a look at after that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94362346
@srowen
I tried setting httpclient in spark-parent's dependencyManagement and removing my direct dependency, but I encounter the same problem as before (httpclient version mismatch, 4.2.5 jar appears in lib_managed). I notice Jackson-mapper-asl is already a managed dependency, but it still caused versioning issues, so it doesn't seem like dependencyManagement is working as intended?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94750550
I don't see anything in `lib_managed` except the datanucleus jars. Jackson should not be something that has to be managed specially like this and I see no reference to Jackson jars in this dir in the code? are you sure this isn't something left around from some other process?
Right now, `mvn -Phadoop-2.4 dependency:tree` shows a dependency on Jackson 1.9.13 only, across the project. What are you seeing that is different and that causes a problem? The current Hadoop deps do not bring in a different version, and Tachyon doesn't add new Hadoop deps (right?)
If Tachyon requires HttpClient 4.3.2+, then that will need to be resolved here. Your change does not actually cause Spark to use 4.3.2, as I say above. That's easy to address, and needs to happen in this PR. We don't need a separate PR for the Kinesis change since that's logically part of the change you will have to make here to make this do what you want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94859092
@srowen
These issues only occur when running the sbt tests using lower versions of Hadoop (ie. 1.0.4).
Tachyon does not require httpclient 4.3.2, it's the selenium tests that do. To be clear, the dependency changes I am making are to address the jar conflicts that occur when running the sbt tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89833019
[Test build #29731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29731/consoleFull) for PR 5354 at commit [`a3a29da`](https://github.com/apache/spark/commit/a3a29da67ef7926f80bc4dcb410e090e0d6f7cb9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89832486
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95680725
It seems like the Jackson dep has to be excluded to get SBT + Hadoop 1.0.4 to work. I think that has to stay then, yeah. I think the httpclient stuff can be cleaned up a small bit but that too is essential.
I'm getting worried at how much the divergence between SBT and Maven is causing us to hack the build, making it harder to get the build right for both. For example, these changes aren't necessary at all for Maven. It's exacerbated by trying to support Hadoop 1.x.
Still maybe we kick this can down the road a bit longer, to get in this change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94995501
@srowen
You shouldn't be able to reproduce a failure with master branch, but upgrading Tachyon triggers the issue (picking up the wrong jar out of the ones that get pulled into lib_managed). This is why I made modifications to prevent the wrong jars from being pulled in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94431123
@srowen I am not sure I understand, where did you see hardcoded versions for httpclient/httpcore ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92215817
The change LGTM but I'll leave it to @pwendell to OK the pom changes. I'm not sure if there's a better way to express basically "don't include any of this guy's transitive dependencies".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-96008364
@srowen @aniketbhatnagar
Thanks for reviewing this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92518143
Sorry for the bad example, here is (hopefully) a better explanation:
Previously:
`tachyon-client` depends on `tachyon`
`tachyon` depends on `hadoop-client`, `curator`, etc.
`tachyon` shades `thrift`
`thrift` depends on `httpclient`
Unfortunately in this scenario, importing `tachyon-client` would only give visibility to `tachyon`'s transitive dependencies `hadoop-client`, `curator`, etc. but not `httpclient` which caused a conflict.
Now:
`tachyon-client` depends on `tachyon`
`tachyon` depends on `hadoop-client`, `curator`, etc.
`tachyon` shades `thrift`
`tachyon` promotes dependencies, causing top level dependencies of `httpclient`, `hadoop-core`, etc. which are now visible and can be excluded.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89850060
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29731/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5354
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92293857
Yea, this is really messy. Why are so many exclusions needed? these are artifacts that are already in the assembly, so, there's no particular problem.
However it does look like the Tachyon client has a lot of big dependencies. Is the client really pulling in `hadoop-core` and `jetty-*`? can the client's dependencies be trimmed somewhat? Basically this declaration says that fewof its dependencies are actually needed.
At the least we need to keep out hadoop-core, the servlet APIs, EL, probably hsqldb if possible, Glassfish. Anything that's added needs its license checked.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94203515
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30528/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94333039
Does dependency management work for transitive dependencies? I'm trying to ensure the higher version wins out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-90249386
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95830293
[Test build #30915 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30915/consoleFull) for PR 5354 at commit [`0eefe4d`](https://github.com/apache/spark/commit/0eefe4d46c0a42859b8c9c0bc0ff98a0beeb440a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **removes the following dependencies:**
* `RoaringBitmap-0.4.5.jar`
* `activation-1.1.jar`
* `akka-actor_2.10-2.3.4-spark.jar`
* `akka-remote_2.10-2.3.4-spark.jar`
* `akka-slf4j_2.10-2.3.4-spark.jar`
* `aopalliance-1.0.jar`
* `arpack_combined_all-0.1.jar`
* `avro-1.7.7.jar`
* `breeze-macros_2.10-0.11.2.jar`
* `breeze_2.10-0.11.2.jar`
* `chill-java-0.5.0.jar`
* `chill_2.10-0.5.0.jar`
* `commons-beanutils-1.7.0.jar`
* `commons-beanutils-core-1.8.0.jar`
* `commons-cli-1.2.jar`
* `commons-codec-1.10.jar`
* `commons-collections-3.2.1.jar`
* `commons-compress-1.4.1.jar`
* `commons-configuration-1.6.jar`
* `commons-digester-1.8.jar`
* `commons-httpclient-3.1.jar`
* `commons-io-2.1.jar`
* `commons-lang-2.5.jar`
* `commons-lang3-3.3.2.jar`
* `commons-math-2.1.jar`
* `commons-math3-3.4.1.jar`
* `commons-net-2.2.jar`
* `compress-lzf-1.0.0.jar`
* `config-1.2.1.jar`
* `core-1.1.2.jar`
* `curator-client-2.4.0.jar`
* `curator-framework-2.4.0.jar`
* `curator-recipes-2.4.0.jar`
* `gmbal-api-only-3.0.0-b023.jar`
* `grizzly-framework-2.1.2.jar`
* `grizzly-http-2.1.2.jar`
* `grizzly-http-server-2.1.2.jar`
* `grizzly-http-servlet-2.1.2.jar`
* `grizzly-rcm-2.1.2.jar`
* `groovy-all-2.3.7.jar`
* `guava-14.0.1.jar`
* `guice-3.0.jar`
* `hadoop-annotations-2.2.0.jar`
* `hadoop-auth-2.2.0.jar`
* `hadoop-client-2.2.0.jar`
* `hadoop-common-2.2.0.jar`
* `hadoop-hdfs-2.2.0.jar`
* `hadoop-mapreduce-client-app-2.2.0.jar`
* `hadoop-mapreduce-client-common-2.2.0.jar`
* `hadoop-mapreduce-client-core-2.2.0.jar`
* `hadoop-mapreduce-client-jobclient-2.2.0.jar`
* `hadoop-mapreduce-client-shuffle-2.2.0.jar`
* `hadoop-yarn-api-2.2.0.jar`
* `hadoop-yarn-client-2.2.0.jar`
* `hadoop-yarn-common-2.2.0.jar`
* `hadoop-yarn-server-common-2.2.0.jar`
* `ivy-2.4.0.jar`
* `jackson-annotations-2.4.0.jar`
* `jackson-core-2.4.4.jar`
* `jackson-core-asl-1.8.8.jar`
* `jackson-databind-2.4.4.jar`
* `jackson-jaxrs-1.8.8.jar`
* `jackson-mapper-asl-1.8.8.jar`
* `jackson-module-scala_2.10-2.4.4.jar`
* `jackson-xc-1.8.8.jar`
* `jansi-1.4.jar`
* `javax.inject-1.jar`
* `javax.servlet-3.0.0.v201112011016.jar`
* `javax.servlet-3.1.jar`
* `javax.servlet-api-3.0.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
* `jcl-over-slf4j-1.7.10.jar`
* `jersey-client-1.9.jar`
* `jersey-core-1.9.jar`
* `jersey-grizzly2-1.9.jar`
* `jersey-guice-1.9.jar`
* `jersey-json-1.9.jar`
* `jersey-server-1.9.jar`
* `jersey-test-framework-core-1.9.jar`
* `jersey-test-framework-grizzly2-1.9.jar`
* `jets3t-0.7.1.jar`
* `jettison-1.1.jar`
* `jetty-util-6.1.26.jar`
* `jline-0.9.94.jar`
* `jline-2.10.4.jar`
* `jodd-core-3.6.3.jar`
* `json4s-ast_2.10-3.2.10.jar`
* `json4s-core_2.10-3.2.10.jar`
* `json4s-jackson_2.10-3.2.10.jar`
* `jsr305-1.3.9.jar`
* `jtransforms-2.4.0.jar`
* `jul-to-slf4j-1.7.10.jar`
* `kryo-2.21.jar`
* `log4j-1.2.17.jar`
* `lz4-1.2.0.jar`
* `management-api-3.0.0-b012.jar`
* `mesos-0.21.0-shaded-protobuf.jar`
* `metrics-core-3.1.0.jar`
* `metrics-graphite-3.1.0.jar`
* `metrics-json-3.1.0.jar`
* `metrics-jvm-3.1.0.jar`
* `minlog-1.2.jar`
* `netty-3.8.0.Final.jar`
* `netty-all-4.0.23.Final.jar`
* `objenesis-1.2.jar`
* `opencsv-2.3.jar`
* `oro-2.0.8.jar`
* `paranamer-2.6.jar`
* `parquet-column-1.6.0rc3.jar`
* `parquet-common-1.6.0rc3.jar`
* `parquet-encoding-1.6.0rc3.jar`
* `parquet-format-2.2.0-rc1.jar`
* `parquet-generator-1.6.0rc3.jar`
* `parquet-hadoop-1.6.0rc3.jar`
* `parquet-jackson-1.6.0rc3.jar`
* `protobuf-java-2.4.1.jar`
* `protobuf-java-2.5.0-spark.jar`
* `py4j-0.8.2.1.jar`
* `pyrolite-2.0.1.jar`
* `quasiquotes_2.10-2.0.1.jar`
* `reflectasm-1.07-shaded.jar`
* `scala-compiler-2.10.4.jar`
* `scala-library-2.10.4.jar`
* `scala-reflect-2.10.4.jar`
* `scalap-2.10.4.jar`
* `scalatest_2.10-2.2.1.jar`
* `slf4j-api-1.7.10.jar`
* `slf4j-log4j12-1.7.10.jar`
* `snappy-java-1.1.1.7.jar`
* `spark-bagel_2.10-1.4.0-SNAPSHOT.jar`
* `spark-catalyst_2.10-1.4.0-SNAPSHOT.jar`
* `spark-core_2.10-1.4.0-SNAPSHOT.jar`
* `spark-graphx_2.10-1.4.0-SNAPSHOT.jar`
* `spark-launcher_2.10-1.4.0-SNAPSHOT.jar`
* `spark-mllib_2.10-1.4.0-SNAPSHOT.jar`
* `spark-network-common_2.10-1.4.0-SNAPSHOT.jar`
* `spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar`
* `spark-repl_2.10-1.4.0-SNAPSHOT.jar`
* `spark-sql_2.10-1.4.0-SNAPSHOT.jar`
* `spark-streaming_2.10-1.4.0-SNAPSHOT.jar`
* `spire-macros_2.10-0.7.4.jar`
* `spire_2.10-0.7.4.jar`
* `stax-api-1.0.1.jar`
* `stream-2.7.0.jar`
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
* `uncommons-maths-1.2.2a.jar`
* `unused-1.0.0.jar`
* `xmlenc-0.52.jar`
* `xz-1.0.jar`
* `zookeeper-3.4.5.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92532627
That ship may have sailed for better or worse. Yes we have to be careful about bringing things into core, so I'm glad to see the exclusions, but I think there are simpler and less brittle ways.
In general if two components need to agree on a particular version of a dependency, that is resolved in `dependencyManagement` in the parent. `exclusions` is not a great way to do it; as soon as, maybe, the other components stops using X that Tachyon needs, now there are 0 copies of it. So that's still the question -- if there are conflicts, what are they, and why not manage the dependency directly?
The `hadoop-core` stuff is maybe more cosmetic. If `tachyon-client` works fine without it, then perhaps it and other dependencies from upstream are merely redundant? at the least, I'd be sort of surprised if lots of these non-client, server-side libraries are _directly_ used by the client. This seems like maybe it's also trying to exclude a bunch of transient dependencies of something already excluded.
You could say that the exclusions really don't hurt much, or if they do we know it quickly, but I think it's worth double-checking the above first, as this exclusion declaration looks surprising. Goodness knows it is necessary in some cases with Hadoop stuff though -- usually to exclude the *same* code under *different artifact names* because upstream projects made uber-jars or shaded. (Looking at you, Jetty)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92540562
For the first point, the conflict was in `httpclient` which was not resolved correctly by the parent, leading to this iteration of the PR.
For the second point, I think all the exclusions added actually do exclude a dependency, at least from a quick look at sbt's dependency-tree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94994768
I'm running SBT tests now just to see if I can reproduce a failure. I guess I'd be surprised if the default SBT build doesn't work, since it would have been this way for a while. I do see that the SBT build adds a bunch of stuff to `lib_managed`, yes, which Maven doesn't.
So, if the dependency changes are not related to Tachyon, I think I'd skip them. There's no problem with Maven right now. The Jackson exclusion should not make a difference; in Maven, the version is consistently 1.8.8 for pre-Hadoop-2 builds, and 1.9.13 for Hadoop 2.2+ builds. The HttpClient situation is a little bit wrong but if it's not actually affecting Tachyon, that can be fixed separately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94869873
The conflicting dependencies were already included prior to my changes. I think since the correct jars were also available, there is a chance to use those and mask the problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95145249
Yeah, that still confuses me. If Tachyon doesn't touch `hadoop-client` or `hadoop-common` I'm not sure how it could change the dependencies. I'm wary of doing things like excluding deps to manage versions, especially when Hadoop does in fact need Jackson.
I can appreciate that -- whatever is going on -- we still want the SBT build to work even if it's not the main build, and still want Hadoop 1.x builds to work even if it's not the default. Let me try applying this change and seeing the diff in SBT myself, to try to verify what is essential to change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95291015
I'm still working through this -- I do see the same problem you see with the httpclient library, and it's because the SBT resolution rules are different. It's weird, but yeah that probably has to be patched up as you have done. I have a proposed change that touches up the handling in Kinesis, etc.
I didn't see a Jackson-related problem, not yet, but that may be masked by earlier failures. Basically, I'd be surprised if this exclusion is the only possible solution, so I want to try it out in a different environment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94191887
[Test build #30528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30528/consoleFull) for PR 5354 at commit [`7c00dfa`](https://github.com/apache/spark/commit/7c00dfa1646fd2bb9e75271980f2f606e9e97301).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94203512
[Test build #30528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30528/consoleFull) for PR 5354 at commit [`7c00dfa`](https://github.com/apache/spark/commit/7c00dfa1646fd2bb9e75271980f2f606e9e97301).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `tachyon-0.6.4.jar`
* `tachyon-client-0.6.4.jar`
* This patch **removes the following dependencies:**
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92488435
@srowen
What I mean by promoting dependencies is like:
Project A depends on Project B
Project B depends on Project C
If Project S depends on Project A, it will pull in A, B, and C (and B & C can be excluded).
If Project A shades project B, Project S cannot see Project B & C to exclude.
If Project A shades project B and promotes transitive dependencies, Project S can see project C to exclude because Project A will just depend on Project C directly (which is why there are so many dependencies).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by aniketbhatnagar <gi...@git.apache.org>.
Github user aniketbhatnagar commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95819955
+1 from my side. having a consistent httpclient version would be so much better!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92459698
@srowen Thanks for the feedback!
The number of dependencies increased because transitive dependencies were promoted to avoid issues with being unable to exclude transitive dependencies from shaded artifacts (httpclient from thrift).
Are you concerned with the size of the exclusion block or just wondering if all the exclusions are necessary?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89472868
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29697/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94329327
I dont think it should be a direct dependency just for this purpose, no. This is what the dependencyManagement section is for. What are you trying to achieve with this? Just ensure a certain version wins out right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94191765
That sure looks cleaner. Although updating the HTTP client version has some implications across the code base -- we need to see if we harmonize correctly on this version and look at the list of changes to decide whether there are possible conflicts -- I suspect that can be OK and that it helps more than hurts.
Still on Jackson: is it that you want to standardize on a certain version? what version? managing the version is a more robust way to do this.
(Reminds me: separately, I should figure out whether we can pull up all of these messy exclusions into the parent. They shouldn't go in the child POMs.)
Does the new release of Tachyon remove some dependencies? that is, it's great that almost all the exclusions are gone, but that doesn't mean that this now brings in more dependencies does it? Jenkins will tell us ---
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95289160
@srowen
Are you completely opposed to using an exclusion for the jackson library? Thanks for taking a look at the sbt build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95830329
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30915/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92468412
I think I am wondering two things -- why is it necessary to include some of these things like commons-beanutils, when they are already part of the dependency graph of even core? and second, I wonder why the tachyon _client_ has to bring in things as large as _hadoop-core_ for example?
(As an aside, the dependency graph should be managed in the parent, so that the exclusions would not have to be repeated if this were used in other modules.)
What do you mean about promoting deps to exclude them? That's the point I'm probably missing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94350049
Should do; give it a try. This should be how you control the version of an artifact that is used without creating an actual dependency on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95292772
@srowen
You can try running org.apache.spark.mllib.regression.RidgeRegressionSuite with hadoop version 1.0.4 to reproduce the Jackson library conflict.
What kind of solutions are you more open to?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94327839
@srowen
Thanks for taking another look!
For the Jackson library, spark already has a dependency managed version, but for transitive dependencies I need to exclude it or introduce it as a top direct dependency, is the latter preferred?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95532097
This is the best I've got, which still works with SBT:
https://github.com/srowen/spark/commit/c1e40f80d11b20546acac6a96f28613cdc1d9fc7
It's mostly the same, but I don't find a jackson exclusion is needed, and I think the httpclient situation could be further tightened.
@aniketbhatnagar @ScrapCodes does the change to the `kinesis-asl` profile make sense? basically, now we need to manage httpclient versions correctly for the whole project, so I figure there's no need to redundantly manage it in the profiles.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92531201
In the original PR, the dependency management was left to maven but it caused dependency issues in tests which is why this new method is being proposed.
For the last question, as you mentioned its a bit off topic for this PR, so let's discuss that in the appropriate forum.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94992672
@srowen
Do you have any other comments? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-95924132
LGTM. Thank you for your perseverance. This gets the change in with minimal additional change to the build, keeps everything compiling and actually improves the management of one dependency along the way.
I think the large list of removed dependencies above is a false positive. It can't remove these.
Let me merge and let's double check that the other Jenkins builds are still happy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94431531
Oh, I'm referring to `commons.httpclient.version` declared in the parent POM. It's actually only used, however, to manage the version of httpclient used by Kinesis. I saw you were the one that added the line, but this could be a false alarm; maybe you were just moving code.
I think the right-er thing to do given the facts here is just make the whole project actually use the version of httpclient / httpcore that this string implies, and make it 4.3.6.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-90277287
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29753/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92512638
Yes, let's call the shaded B artifact B'. Are you saying that the published artifact of A shades B and C? yes, then it contains B' and C' and they are inseparable. It isn't usual to publish a shaded artifact like that unless it is an assembly / uber jar, which is not what projects depend on when consuming via Maven.
A would still show a dependency on B and C in the POM, at least, if it were shaded at assembly time using something like the shade plugin.
You are saying that A shades B and C but then declares C as a direct, separate dependency? yes, then it shows a direct dependency on C, though I claim it already had a transitive dependency on C in Maven. We may still be talking about different things when it comes to what is published.
But, what is A here, `tachyon-client`? I'm not sure it's relevant to what I am getting at. If C includes `commons-beanutils`, why exclude it? and, why does C include `hadoop-core` for example? Again, if I understand what's what here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94191767
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94191773
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by haoyuan <gi...@git.apache.org>.
Github user haoyuan commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89439800
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89421183
[Test build #29691 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29691/consoleFull) for PR 5354 at commit [`0ae6c97`](https://github.com/apache/spark/commit/0ae6c97f97ce0e5f3b8e51ddbdfa39f33069a5c7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `com.sun.el-1.0.0.v201105211818.jar`
* `commons-el-1.0.jar`
* `commons-logging-1.0.3.jar`
* `hadoop-core-1.0.4.jar`
* `hsqldb-1.8.0.10.jar`
* `javax.el-2.1.0.v201105211819.jar`
* `javax.servlet.jsp-2.1.0.v201105211820.jar`
* `javax.servlet.jsp.jstl-1.2.0.v201105211821.jar`
* `jetty-continuation-8.1.14.v20131031.jar`
* `jetty-http-8.1.14.v20131031.jar`
* `jetty-io-7.6.15.v20140411.jar`
* `jetty-security-8.1.14.v20131031.jar`
* `jetty-util-8.1.14.v20131031.jar`
* `jetty-xml-7.6.15.v20140411.jar`
* `org.apache.jasper.glassfish-2.1.0.v201110031002.jar`
* `org.apache.taglibs.standard.glassfish-1.2.0.v201112081803.jar`
* `org.eclipse.jdt.core-3.7.1.jar`
* `tachyon-0.6.3.jar`
* `tachyon-client-0.6.3.jar`
* This patch **removes the following dependencies:**
* `tachyon-0.5.0.jar`
* `tachyon-client-0.5.0.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89393128
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by haoyuan <gi...@git.apache.org>.
Github user haoyuan commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-90203297
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94178568
@JoshRosen @srowen @aarondav
I've updated the PR to resolve the http client issue by upgrading the version and specifying the version in sql/hive (which was pulling in the 4.2.x version).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94432612
hm.. your approach sounds good to me, we should really make sure we have the same version of a library across dependent projects.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by calvinjia <gi...@git.apache.org>.
Github user calvinjia commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-94530456
@srowen
Despite Jackson being managed by dependencyManagement, lower versions make it into managed_libs which cause issues when `mllib` tests run, since they expect the higher version. This is why I exclude the library from hadoop-client which brings in the lower versions.
For `httpclient`, `4.2.5` is brought in by `sql/hive` when they reference `libthrift`. This causes selenium tests to break since it expects `4.3.2`.
Finding the perfect way to reconcile Spark's dependencies as you mentioned before, deserves its own ticket and will require substantial effort. For example, it is not even easy to tell that different versions of `httpclient` are referenced in Spark.
@ScrapCodes
Let's address cleaning up the Kinesis profile in a separate PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89421203
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29691/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by haoyuan <gi...@git.apache.org>.
Github user haoyuan commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89403342
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-89404274
[Test build #29691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29691/consoleFull) for PR 5354 at commit [`0ae6c97`](https://github.com/apache/spark/commit/0ae6c97f97ce0e5f3b8e51ddbdfa39f33069a5c7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6122][Core] Upgrade tachyon-client vers...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5354#issuecomment-92521916
In this case, `tachyon` includes `thrift` (shaded) and `httpclient` (unshaded). Adding a direct dependency on `httpclient` shouldn't do anything. Shading hasn't changed that it (unshaded) is a transitive dependency. Well, maybe there's more to it.
But my point remains something else. If `httpclient` is needed, you can't exclude it, right? If Spark also depends on `httpclient`, that's fine: Maven resolves the conflict. If Maven's choice is wrong, then _Spark_ manages the dependency in `dependencyManagement`, but, this isn't done with exclusions. For example, this is the situation with `commons-beanutils`.
I suppose my secondary point is still that `tachyon-client` seems to pull in lots of stuff that don't seem like things a client-side library should need, like a web server or `hadoop-core`. I know that this is at best a Tachyon question, but, maybe still valid to consider here since Spark is a primary consumer of Tachyon.
(Third question, which is off topic here I suppose is, why does Tachyon belong as a dependency of the core anyway? I suppose you could make the same argument for Hadoop of course, but I didn't see Tachyon as so nearly essential. I would have expected an optional module or something like YARN, since it's not something that will be enabled by most users.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org