You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by wangyum <gi...@git.apache.org> on 2018/09/04 08:42:54 UTC
[GitHub] spark pull request #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
GitHub user wangyum opened a pull request:
https://github.com/apache/spark/pull/22327
[SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
## What changes were proposed in this pull request?
Revert Hadoop 2.7 to 2.7.3 to fix permission issue. The issue occurred in this commit: https://github.com/apache/hadoop/commit/feb886f2093ea5da0cd09c69bd1360a335335c86
## How was this patch tested?
unit tests and manual tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangyum/spark SPARK-25330
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22327.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22327
----
commit f89448b7b0598a59f750a324e869e7768cfedbc1
Author: Yuming Wang <yu...@...>
Date: 2018-09-04T08:31:13Z
Revert Hadoop 2.7 to 2.7.3
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22327: [SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2....
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22327
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/22327
The goal for the 2.7.x line should be "nothing breaks", which is precisely why it's only getting critical patches. Reverting might make the problem go away, but you can assume that everyone running HDFS clusters with something based on the 2.7.x line will get this patch in before long: it's best to identify what's up and address it
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/22327
cc @srowen @steveloughran
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22327
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22327
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95664/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22327
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2831/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/22327
The 2.7.x branches updates are generally all security plus some fixes for JVM/JDK regressions.
without looking at the details, you can assume that the regression will be related to one of these and for that reason, wouldn't recommend rolling back. Better to find the problem and come up with a fix or workaround.
Created [HADOOP-15722](https://issues.apache.org/jira/browse/HADOOP-15722) to cover this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/22327
How about revert it to branch-2.3 as we are going to release 2.3.2?
We have time to fix it before releasing 2.4.0.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22327
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2.7 to 2....
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22327
From further discussion on the JIRA, looks like there is not an obvious immediate resolution, and we are about to release 2.3.2 and 2.4, and have a nontrivial problem vs a theoretical set of bug fixes in 2.7.7. I'd say roll back now. If we do figure something out soon we can update again
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22327
If it's an unintentional regression in Hadoop, and is impacting users, I'd agree with down-grading the version here (at least to the version that doesn't exhibit the issue).
If it were intentional behavior, we'd really need to work around it.
@steveloughran seems like you think it's a regression insofar as behavior changes like this aren't expected in maintenance releases?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2.7 to 2....
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22327
Merged to master/2.4/2.3
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22327
**[Test build #95664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95664/testReport)** for PR 22327 at commit [`f89448b`](https://github.com/apache/spark/commit/f89448b7b0598a59f750a324e869e7768cfedbc1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22327
I think that is reasonable personally
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/22327
Hm is this really a Hadoop thing? Looks like a local env issue? Not sure what may have changed or be different about the temp dir here.
Users are going to use this with Hadoop 2.7.7+; I wonder if we face the issue regardless.
Yes over to @steveloughran for any thoughts on what this may be or if it is a known issue.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/22327
Yes. This is a Hadoop thing. I try to build Hadoop 2.7.7 with [`Configuration.getRestrictParserDefault(Object resource)`](https://github.com/apache/hadoop/blob/release-2.7.7-RC0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L236) = true and false.
It succeeded when `Configuration.getRestrictParserDefault(Object resource)=false`, but failed when `Configuration.getRestrictParserDefault(Object resource)=true`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22327
**[Test build #95664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95664/testReport)** for PR 22327 at commit [`f89448b`](https://github.com/apache/spark/commit/f89448b7b0598a59f750a324e869e7768cfedbc1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org