You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pwendell <gi...@git.apache.org> on 2014/05/04 06:54:32 UTC
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
GitHub user pwendell opened a pull request:
https://github.com/apache/spark/pull/627
SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7.
This add some guards and good warning messages if users hit this issue. /cc @aarondav with whom I discussed parts of the design.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwendell/spark jdk6
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/627.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #627
----
commit 94e9f84668b40edf539b0f5c0ffa271b0090c2a7
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-05-04T04:52:44Z
SPARK-1703 Warn users if Spark is run on JRE6 but compiled with JDK7.
This add some guards and good warning messages if users hit this issue.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42124219
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42124220
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42124759
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14639/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42141903
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14646/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42140723
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by dkanoafry <gi...@git.apache.org>.
Github user dkanoafry commented on a diff in the pull request:
https://github.com/apache/spark/pull/627#discussion_r12326286
--- Diff: make-distribution.sh ---
@@ -50,6 +50,20 @@ if [ $? == -1 ] ;then
exit -1;
fi
+if [ -z "${JAVA_HOME}" ]; then
+ echo "Error: JAVA_HOME is not set, cannot proceed."
+ exit -1
+fi
+
+JAVA_CMD=$JAVA_HOME/bin/java
+JAVA_VERSION=$($JAVA_CMD -version 2>&1)
+if ! [[ "$JAVA_VERSION" =~ "1.6" ]]; then
--- End diff --
require JDK 1.6?? this doesn't seem to be in line with the spirit of SPARK-1703
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42142082
Thanks, meregd.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/627#discussion_r12260338
--- Diff: bin/spark-class ---
@@ -138,7 +138,14 @@ if [ -e "$TOOLS_DIR"/target/spark-tools*[0-9Tg].jar ]; then
fi
# Compute classpath using external script
-CLASSPATH=`$FWDIR/bin/compute-classpath.sh`
+CLASSPATH_OUTPUT=$($FWDIR/bin/compute-classpath.sh)
--- End diff --
can we go with lower case variable names for "local" variables?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/627#discussion_r12263308
--- Diff: bin/spark-class ---
@@ -138,7 +138,14 @@ if [ -e "$TOOLS_DIR"/target/spark-tools*[0-9Tg].jar ]; then
fi
# Compute classpath using external script
-CLASSPATH=`$FWDIR/bin/compute-classpath.sh`
+CLASSPATH_OUTPUT=$($FWDIR/bin/compute-classpath.sh)
--- End diff --
I also like that style, but this class uses the other style (always uses all-caps) pervasively. I can change it though, will just become inconsistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/627#discussion_r12260344
--- Diff: make-distribution.sh ---
@@ -50,6 +50,20 @@ if [ $? == -1 ] ;then
exit -1;
fi
+if [ -z "${JAVA_HOME}" ]; then
+ echo "Error: JAVA_HOME is not set, cannot proceed."
+ exit -1
+fi
+
+JAVA_CMD=$JAVA_HOME/bin/java
+JAVA_VERSION=$($JAVA_CMD -version 2>&1)
+if ! [[ "$JAVA_VERSION" =~ "1.6" ]]; then
+ echo "Error: Java version was not 1.6. Spark must be compiled with JDK 1.6 "
--- End diff --
maybe add a comment on how to fix this (i.e., set JAVA_HOME).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42141902
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42141371
@aarondav addressed your comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/627
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/627#discussion_r12289304
--- Diff: bin/compute-classpath.sh ---
@@ -54,6 +60,14 @@ else
else
ASSEMBLY_JAR=`ls "$ASSEMBLY_DIR"/spark-assembly*hadoop*.jar`
fi
+ jar_error_check=$($JAR_CMD -tf $ASSEMBLY_JAR org/apache/spark/SparkContext 2>&1)
+ if [[ "$jar_error_check" =~ "invalid CEN header" ]]; then
+ echo "Loading Spark jar with '$JAR_CMD' failed. "
+ echo "This is likely because Spark was compiled with Java 7 and run "
+ echo "with Java 6. (see SPARK-1703). Please use Java 7 to run Spark "
+ echo "or build Spark with Java 6."
+ exit 1
+ fi
--- End diff --
I realize this is already merged, but it looks like the jar error check is only tested on the assembly jar (not on the reps assembly jar). It might be good to check it in both cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42124758
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42140732
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42141941
LGTM. Merge at will.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1703 Warn users if Spark is run on JRE6 ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/627#issuecomment-42127232
Two very minor comments, otherwise LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---