You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by punya <gi...@git.apache.org> on 2015/04/07 03:34:18 UTC
[GitHub] spark pull request: Bump version of apache commons-math3
GitHub user punya opened a pull request:
https://github.com/apache/spark/pull/5380
Bump version of apache commons-math3
Version 3.1.1 is two years old and the newer version includes
approximate percentile statistics (among other things).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/punya/spark-1 patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5380.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5380
----
commit 226622bf8aca5c7c6e3ddb61c00d78f7b082e063
Author: Punyashloka Biswal <pu...@gmail.com>
Date: 2015-04-07T01:33:57Z
Bump version of apache commons-math3
Version 3.1.1 is two years old and the newer version includes
approximate percentile statistics (among other things).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-92151126
I've tried Hadoop 1.2.1, 2.2.0 and 2.6.0 and tests passed. I am fairly sure this is an OK update. Any other experience on your end?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90858684
My guess is that this is probably OK -- I have actually been using commons math 3.4.1 in a project on Hadoop 2.6 without issues, though I recall a subtle and nasty incompatibility problem from 3.2 to 3.3 (that is very unlikely to be relevant here or manifest here). Clearly the intent has been backwards compatibility: http://commons.apache.org/proper/commons-math/changes-report.html
Although it's always safe to try to harmonize dependencies, it's not 100% possible. If there's a compelling reason to update, and we see that tests pass in a few permutations of Hadoop builds, then I'd support it. For example can you try `mvn [opts] -DskipTests clean package; mvn [opts] test` for options like `-Phadoop-2.2` and `-Phadoop-2.4 -Dhadoop.version=2.6.0`? if those pass that gives more confidence that there's no issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-92343655
Unfortunately, I got random flaky tests when I ran it with 2.6.0 (and I didn't get a chance to determine whether the flakes were associated to the version update). My hunch is that they weren't (because they had nothing to do with commons-math3 and seemed related to timeouts). Based on your experience I'd favor merging the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Bump version of apache commons-math3
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90318628
@punya can you create a Jira ticket for this and put it in the PR subject like "[SPARK-12345] Bump version of apache commons-math3" ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90337315
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29765/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90337297
[Test build #29765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29765/consoleFull) for PR 5380 at commit [`226622b`](https://github.com/apache/spark/commit/226622bf8aca5c7c6e3ddb61c00d78f7b082e063).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `commons-math3-3.4.1.jar`
* This patch **removes the following dependencies:**
* `commons-math3-3.1.1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Bump version of apache commons-math3
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90307697
[Test build #29765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29765/consoleFull) for PR 5380 at commit [`226622b`](https://github.com/apache/spark/commit/226622bf8aca5c7c6e3ddb61c00d78f7b082e063).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-93098262
@mengxr Hm! that's good to know. I agree that I doubt users are affected by this -- Spark isn't -- and users should probably be packaging their own copy of `commons-math3` if it matters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-91236980
Thanks for the feedback, I'll let you know how the tests go.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90617620
@srowen based on http://central.maven.org/maven2/org/apache/hadoop/hadoop-core/0.20.2/hadoop-core-0.20.2.pom it looks like hadoop (at least this version) doesn't depend on `commons-math3` at all. Perhaps we should annotate dependencies that are pinned based on Hadoop profiles to know what needs to be checked when we want to upgrade something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-93090902
@punya @srowen You can use japi-compliance-checker to check binary/source compatibility. I compared 3.1.1 and 3.4.1, and there are some break changes in the `geometry` package. I'm okay with this change, but I don't really know how many users are using the `geometry` package.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90511486
It's not entirely that simple, since this is meant to match the version used in other dependencies like Hadoop. The Commons Math3 dependencies aren't 100% compatible across minor releases. I generally favor this update, but you would first need to run tests with a number of different Hadoop profiles, for example, to verify it still works. Give that a go first?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5380
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90640336
I looked at https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-project/2.6.0/hadoop-project-2.6.0.pom and did find that they depend on commons-math3 version 3.1.1. Do we aim for strict compatibility with the versions in the Hadoop parent POM, or do we only consider a subset of Hadoop projects that might be on the classpath with Spark?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-92763728
I think we'll want to upgrade this at some point since there are a number of bug fixes and small new features we want to use. I think I've convinced myself that this is empirically compatible, intended to be compatible, and the particular issue I got hit by is not relevant here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6731] Bump version of apache commons-ma...
Posted by punya <gi...@git.apache.org>.
Github user punya commented on the pull request:
https://github.com/apache/spark/pull/5380#issuecomment-90560713
Is there an automated way to do that?
On Tue, Apr 7, 2015 at 7:24 AM Sean Owen <no...@github.com> wrote:
> It's not entirely that simple, since this is meant to match the version
> used in other dependencies like Hadoop. The Commons Math3 dependencies
> aren't 100% compatible across minor releases. I generally favor this
> update, but you would first need to run tests with a number of different
> Hadoop profiles, for example, to verify it still works. Give that a go
> first?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5380#issuecomment-90511486>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org