You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2014/01/07 09:18:31 UTC
[01/14] git commit: Add a script to download sbt if not present on
the system
Updated Branches:
refs/heads/master b97ef218f -> a862cafac
Add a script to download sbt if not present on the system
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/9e9a913c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/9e9a913c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/9e9a913c
Branch: refs/heads/master
Commit: 9e9a913c2ffe0b01f051c18018740934a6a8691e
Parents: d43ad3e
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sat Jan 4 20:08:35 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sat Jan 4 20:08:35 2014 -0800
----------------------------------------------------------------------
.gitignore | 1 +
project/build.properties | 2 ++
sbt | 48 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 51 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/9e9a913c/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index 399362f..1692bde 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,6 +4,7 @@
*.iml
*.iws
.idea/
+.sbtlib/*.jar
.settings
.cache
/build/
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/9e9a913c/project/build.properties
----------------------------------------------------------------------
diff --git a/project/build.properties b/project/build.properties
index 9647277..059edac 100644
--- a/project/build.properties
+++ b/project/build.properties
@@ -15,4 +15,6 @@
# limitations under the License.
#
+# Note: If you change the sbt version please also change the sbt download
+# script in the root directory of the project called sbt.
sbt.version=0.12.4
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/9e9a913c/sbt
----------------------------------------------------------------------
diff --git a/sbt b/sbt
new file mode 100755
index 0000000..e04a20c
--- /dev/null
+++ b/sbt
@@ -0,0 +1,48 @@
+#!/bin/bash
+# This script launches sbt for this project. If present it uses the system
+# version of sbt. If there is no system version of sbt it attempts to download
+# sbt locally.
+SBT_VERSION=0.12.4
+URL1=http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
+URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
+JAR=.sbtlib/sbt-launch-${SBT_VERSION}.jar
+
+printf "Checking for system sbt ["
+if hash sbt 2>/dev/null; then
+ printf "FOUND]\n"
+ # Use System SBT
+ sbt
+else
+ printf "NOT FOUND]\n"
+ # Download sbt or use allready downloaded
+ if [ ! -d .sbtlib ]; then
+ mkdir .sbtlib
+ fi
+ if [ ! -f ${JAR} ]; then
+ # Download
+ printf "Attempting to fetch sbt\n"
+ if hash curl 2>/dev/null; then
+ curl --progress-bar ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}
+ elif hash wget 2>/dev/null; then
+ wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
+ else
+ printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
+ exit
+ fi
+ fi
+ if [ ! -f ${JAR} ]; then
+ # We failed to download
+ printf "Our attempt to download sbt locally to {$JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
+ exit
+ fi
+ printf "Launching sbt from .sbtlib\n"
+ java \
+ -Duser.timezone=UTC \
+ -Djava.awt.headless=true \
+ -Dfile.encoding=UTF-8 \
+ -XX:MaxPermSize=256m \
+ -Xmx1g \
+ -noverify \
+ -jar ${JAR} \
+ "$@"
+fi
[13/14] git commit: Use awk to extract the version
Posted by rx...@apache.org.
Use awk to extract the version
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/60a7a6b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/60a7a6b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/60a7a6b3
Branch: refs/heads/master
Commit: 60a7a6b31a441f5f716091316f90819c62b22e70
Parents: b590adb
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Mon Jan 6 23:45:27 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Mon Jan 6 23:45:27 2014 -0800
----------------------------------------------------------------------
project/build.properties | 3 ---
sbt/sbt | 2 +-
2 files changed, 1 insertion(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/60a7a6b3/project/build.properties
----------------------------------------------------------------------
diff --git a/project/build.properties b/project/build.properties
index 03438f6..839f5fb 100644
--- a/project/build.properties
+++ b/project/build.properties
@@ -14,7 +14,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-
-# Note: If you change the sbt version please also change the sbt download
-# script sbt/sbt
sbt.version=0.12.4
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/60a7a6b3/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
index 2c19987..22672f2 100755
--- a/sbt/sbt
+++ b/sbt/sbt
@@ -2,7 +2,7 @@
# This script launches sbt for this project. If present it uses the system
# version of sbt. If there is no system version of sbt it attempts to download
# sbt locally.
-SBT_VERSION=0.12.4
+SBT_VERSION=`awk -F "=" '/sbt\\.version/ {print $2}' ./project/build.properties`
URL1=http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
JAR=sbt/sbt-launch-${SBT_VERSION}.jar
[05/14] git commit: Make sbt in the sbt directory
Posted by rx...@apache.org.
Make sbt in the sbt directory
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/0d6700eb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/0d6700eb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/0d6700eb
Branch: refs/heads/master
Commit: 0d6700eb5a901caad511bed3d21e43f16adcf192
Parents: d2a5c75
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sat Jan 4 21:44:26 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sat Jan 4 21:44:26 2014 -0800
----------------------------------------------------------------------
sbt | 48 ------------------------------------------------
sbt/sbt | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 48 insertions(+), 48 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/0d6700eb/sbt
----------------------------------------------------------------------
diff --git a/sbt b/sbt
deleted file mode 100755
index d21806e..0000000
--- a/sbt
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/bash
-# This script launches sbt for this project. If present it uses the system
-# version of sbt. If there is no system version of sbt it attempts to download
-# sbt locally.
-SBT_VERSION=0.12.4
-URL1=http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
-URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
-JAR=.sbtlib/sbt-launch-${SBT_VERSION}.jar
-
-printf "Checking for system sbt ["
-if hash sbt 2>/dev/null; then
- printf "FOUND]\n"
- # Use System SBT
- sbt $@
-else
- printf "NOT FOUND]\n"
- # Download sbt or use already downloaded
- if [ ! -d .sbtlib ]; then
- mkdir .sbtlib
- fi
- if [ ! -f ${JAR} ]; then
- # Download
- printf "Attempting to fetch sbt\n"
- if hash curl 2>/dev/null; then
- curl --progress-bar ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}
- elif hash wget 2>/dev/null; then
- wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
- else
- printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
- exit
- fi
- fi
- if [ ! -f ${JAR} ]; then
- # We failed to download
- printf "Our attempt to download sbt locally to {$JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
- exit
- fi
- printf "Launching sbt from .sbtlib\n"
- java \
- -Duser.timezone=UTC \
- -Djava.awt.headless=true \
- -Dfile.encoding=UTF-8 \
- -XX:MaxPermSize=256m \
- -Xmx1g \
- -noverify \
- -jar ${JAR} \
- "$@"
-fi
diff --git a/sbt/sbt b/sbt/sbt
new file mode 100755
index 0000000..d21806e
--- /dev/null
+++ b/sbt/sbt
@@ -0,0 +1,48 @@
+#!/bin/bash
+# This script launches sbt for this project. If present it uses the system
+# version of sbt. If there is no system version of sbt it attempts to download
+# sbt locally.
+SBT_VERSION=0.12.4
+URL1=http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
+URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
+JAR=.sbtlib/sbt-launch-${SBT_VERSION}.jar
+
+printf "Checking for system sbt ["
+if hash sbt 2>/dev/null; then
+ printf "FOUND]\n"
+ # Use System SBT
+ sbt $@
+else
+ printf "NOT FOUND]\n"
+ # Download sbt or use already downloaded
+ if [ ! -d .sbtlib ]; then
+ mkdir .sbtlib
+ fi
+ if [ ! -f ${JAR} ]; then
+ # Download
+ printf "Attempting to fetch sbt\n"
+ if hash curl 2>/dev/null; then
+ curl --progress-bar ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}
+ elif hash wget 2>/dev/null; then
+ wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
+ else
+ printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
+ exit
+ fi
+ fi
+ if [ ! -f ${JAR} ]; then
+ # We failed to download
+ printf "Our attempt to download sbt locally to {$JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
+ exit
+ fi
+ printf "Launching sbt from .sbtlib\n"
+ java \
+ -Duser.timezone=UTC \
+ -Djava.awt.headless=true \
+ -Dfile.encoding=UTF-8 \
+ -XX:MaxPermSize=256m \
+ -Xmx1g \
+ -noverify \
+ -jar ${JAR} \
+ "$@"
+fi
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/0d6700eb/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
new file mode 100755
index 0000000..d21806e
--- /dev/null
+++ b/sbt/sbt
@@ -0,0 +1,48 @@
+#!/bin/bash
+# This script launches sbt for this project. If present it uses the system
+# version of sbt. If there is no system version of sbt it attempts to download
+# sbt locally.
+SBT_VERSION=0.12.4
+URL1=http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
+URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
+JAR=.sbtlib/sbt-launch-${SBT_VERSION}.jar
+
+printf "Checking for system sbt ["
+if hash sbt 2>/dev/null; then
+ printf "FOUND]\n"
+ # Use System SBT
+ sbt $@
+else
+ printf "NOT FOUND]\n"
+ # Download sbt or use already downloaded
+ if [ ! -d .sbtlib ]; then
+ mkdir .sbtlib
+ fi
+ if [ ! -f ${JAR} ]; then
+ # Download
+ printf "Attempting to fetch sbt\n"
+ if hash curl 2>/dev/null; then
+ curl --progress-bar ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}
+ elif hash wget 2>/dev/null; then
+ wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
+ else
+ printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
+ exit
+ fi
+ fi
+ if [ ! -f ${JAR} ]; then
+ # We failed to download
+ printf "Our attempt to download sbt locally to {$JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
+ exit
+ fi
+ printf "Launching sbt from .sbtlib\n"
+ java \
+ -Duser.timezone=UTC \
+ -Djava.awt.headless=true \
+ -Dfile.encoding=UTF-8 \
+ -XX:MaxPermSize=256m \
+ -Xmx1g \
+ -noverify \
+ -jar ${JAR} \
+ "$@"
+fi
[07/14] git commit: reindent
Posted by rx...@apache.org.
reindent
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/df92f1c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/df92f1c0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/df92f1c0
Branch: refs/heads/master
Commit: df92f1c0254dc9073c18bc7b76f8b9523ecd7cec
Parents: d7d95a0
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sat Jan 4 21:48:35 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sat Jan 4 21:48:35 2014 -0800
----------------------------------------------------------------------
sbt/sbt | 62 ++++++++++++++++++++++++++++++------------------------------
1 file changed, 31 insertions(+), 31 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/df92f1c0/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
index a7146e3..6d2caca 100755
--- a/sbt/sbt
+++ b/sbt/sbt
@@ -9,40 +9,40 @@ JAR=sbt/sbt-launch-${SBT_VERSION}.jar
printf "Checking for system sbt ["
if hash sbt 2>/dev/null; then
- printf "FOUND]\n"
- # Use System SBT
- sbt $@
+ printf "FOUND]\n"
+ # Use System SBT
+ sbt $@
else
- printf "NOT FOUND]\n"
- # Download sbt or use already downloaded
- if [ ! -d .sbtlib ]; then
- mkdir .sbtlib
- fi
- if [ ! -f ${JAR} ]; then
- # Download
- printf "Attempting to fetch sbt\n"
- if hash curl 2>/dev/null; then
+ printf "NOT FOUND]\n"
+ # Download sbt or use already downloaded
+ if [ ! -d .sbtlib ]; then
+ mkdir .sbtlib
+ fi
+ if [ ! -f ${JAR} ]; then
+ # Download
+ printf "Attempting to fetch sbt\n"
+ if hash curl 2>/dev/null; then
curl --progress-bar ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}
- elif hash wget 2>/dev/null; then
+ elif hash wget 2>/dev/null; then
wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
- else
+ else
printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
exit
- fi
- fi
- if [ ! -f ${JAR} ]; then
- # We failed to download
- printf "Our attempt to download sbt locally to {$JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
- exit
- fi
- printf "Launching sbt from .sbtlib\n"
- java \
- -Duser.timezone=UTC \
- -Djava.awt.headless=true \
- -Dfile.encoding=UTF-8 \
- -XX:MaxPermSize=256m \
- -Xmx1g \
- -noverify \
- -jar ${JAR} \
- "$@"
+ fi
+ fi
+ if [ ! -f ${JAR} ]; then
+ # We failed to download
+ printf "Our attempt to download sbt locally to {$JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
+ exit
+ fi
+ printf "Launching sbt from .sbtlib\n"
+ java \
+ -Duser.timezone=UTC \
+ -Djava.awt.headless=true \
+ -Dfile.encoding=UTF-8 \
+ -XX:MaxPermSize=256m \
+ -Xmx1g \
+ -noverify \
+ -jar ${JAR} \
+ "$@"
fi
[11/14] git commit: CR feedback (sbt -> sbt/sbt and correct JAR path
in script) :)
Posted by rx...@apache.org.
CR feedback (sbt -> sbt/sbt and correct JAR path in script) :)
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/2dc83de7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/2dc83de7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/2dc83de7
Branch: refs/heads/master
Commit: 2dc83de72e7803fbf69f55fd1edf1969bc48e074
Parents: 7d0094b
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sun Jan 5 23:29:26 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sun Jan 5 23:29:26 2014 -0800
----------------------------------------------------------------------
bin/run-example | 2 +-
project/build.properties | 2 +-
sbt/sbt | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/2dc83de7/bin/run-example
----------------------------------------------------------------------
diff --git a/bin/run-example b/bin/run-example
index dfb4bf7..6c5d4a6 100755
--- a/bin/run-example
+++ b/bin/run-example
@@ -55,7 +55,7 @@ if [ -e "$EXAMPLES_DIR"/target/spark-examples*[0-9Tg].jar ]; then
fi
if [[ -z $SPARK_EXAMPLES_JAR ]]; then
echo "Failed to find Spark examples assembly in $FWDIR/examples/target" >&2
- echo "You need to build Spark with sbt assembly before running this program" >&2
+ echo "You need to build Spark with sbt/sbt assembly before running this program" >&2
exit 1
fi
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/2dc83de7/project/build.properties
----------------------------------------------------------------------
diff --git a/project/build.properties b/project/build.properties
index 059edac..03438f6 100644
--- a/project/build.properties
+++ b/project/build.properties
@@ -16,5 +16,5 @@
#
# Note: If you change the sbt version please also change the sbt download
-# script in the root directory of the project called sbt.
+# script sbt/sbt
sbt.version=0.12.4
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/2dc83de7/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
index 0951e9d..af422d9 100755
--- a/sbt/sbt
+++ b/sbt/sbt
@@ -35,7 +35,7 @@ else
printf "Our attempt to download sbt locally to ${JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
exit -1
fi
- printf "Launching sbt from .sbtlib\n"
+ printf "Launching sbt from ${JAR}\n"
java \
-Xmx1200m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=256m \
-jar ${JAR} \
[08/14] git commit: Code review feedback
Posted by rx...@apache.org.
Code review feedback
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d86dc74d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d86dc74d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/d86dc74d
Branch: refs/heads/master
Commit: d86dc74d796121b61ff43c632791c52dd49ff8ad
Parents: df92f1c
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sun Jan 5 22:05:30 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sun Jan 5 22:05:30 2014 -0800
----------------------------------------------------------------------
README.md | 14 ++++++++------
docs/README.md | 4 ++--
docs/_plugins/copy_api_dirs.rb | 4 ++--
docs/api.md | 2 +-
docs/hadoop-third-party-distributions.md | 2 +-
docs/index.md | 6 +++---
docs/python-programming-guide.md | 2 +-
docs/quick-start.md | 8 ++++----
docs/running-on-yarn.md | 6 +++---
docs/scala-programming-guide.md | 2 +-
make-distribution.sh | 7 +++++--
sbt/sbt | 13 ++++---------
12 files changed, 35 insertions(+), 35 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 2c08a4a..b91e4cf 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,9 @@ This README file only contains basic setup instructions.
## Building
Spark requires Scala 2.10. The project is built using Simple Build Tool (SBT),
-which can be obtained [here](http://www.scala-sbt.org). To build Spark and its example programs, run:
+which can be obtained [here](http://www.scala-sbt.org). If SBT is installed we
+will use the system version of sbt otherwise we will attempt to download it
+automatically. To build Spark and its example programs, run:
./sbt/sbt assembly
@@ -55,22 +57,22 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop
versions without YARN, use:
# Apache Hadoop 1.2.1
- $ SPARK_HADOOP_VERSION=1.2.1 sbt assembly
+ $ SPARK_HADOOP_VERSION=1.2.1 sbt/sbt assembly
# Cloudera CDH 4.2.0 with MapReduce v1
- $ SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.2.0 sbt assembly
+ $ SPARK_HADOOP_VERSION=2.0.0-mr1-cdh4.2.0 sbt/sbt assembly
For Apache Hadoop 2.2.X, 2.1.X, 2.0.X, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions
with YARN, also set `SPARK_YARN=true`:
# Apache Hadoop 2.0.5-alpha
- $ SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt assembly
+ $ SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly
# Cloudera CDH 4.2.0 with MapReduce v2
- $ SPARK_HADOOP_VERSION=2.0.0-cdh4.2.0 SPARK_YARN=true sbt assembly
+ $ SPARK_HADOOP_VERSION=2.0.0-cdh4.2.0 SPARK_YARN=true sbt/sbt assembly
# Apache Hadoop 2.2.X and newer
- $ SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true sbt assembly
+ $ SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true sbt/sbt assembly
When developing a Spark application, specify the Hadoop version by adding the
"hadoop-client" artifact to your project's dependencies. For example, if you're
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/README.md
----------------------------------------------------------------------
diff --git a/docs/README.md b/docs/README.md
index e3d6c9a..dfcf753 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -27,10 +27,10 @@ To mark a block of code in your markdown to be syntax highlighted by jekyll duri
## API Docs (Scaladoc and Epydoc)
-You can build just the Spark scaladoc by running `sbt doc` from the SPARK_PROJECT_ROOT directory.
+You can build just the Spark scaladoc by running `sbt/sbt doc` from the SPARK_PROJECT_ROOT directory.
Similarly, you can build just the PySpark epydoc by running `epydoc --config epydoc.conf` from the SPARK_PROJECT_ROOT/pyspark directory.
-When you run `jekyll` in the docs directory, it will also copy over the scaladoc for the various Spark subprojects into the docs directory (and then also into the _site directory). We use a jekyll plugin to run `sbt doc` before building the site so if you haven't run it (recently) it may take some time as it generates all of the scaladoc. The jekyll plugin also generates the PySpark docs using [epydoc](http://epydoc.sourceforge.net/).
+When you run `jekyll` in the docs directory, it will also copy over the scaladoc for the various Spark subprojects into the docs directory (and then also into the _site directory). We use a jekyll plugin to run `sbt/sbt doc` before building the site so if you haven't run it (recently) it may take some time as it generates all of the scaladoc. The jekyll plugin also generates the PySpark docs using [epydoc](http://epydoc.sourceforge.net/).
NOTE: To skip the step of building and copying over the Scala and Python API docs, run `SKIP_API=1 jekyll`.
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/_plugins/copy_api_dirs.rb
----------------------------------------------------------------------
diff --git a/docs/_plugins/copy_api_dirs.rb b/docs/_plugins/copy_api_dirs.rb
index ef9912c..431de90 100644
--- a/docs/_plugins/copy_api_dirs.rb
+++ b/docs/_plugins/copy_api_dirs.rb
@@ -26,8 +26,8 @@ if not (ENV['SKIP_API'] == '1' or ENV['SKIP_SCALADOC'] == '1')
curr_dir = pwd
cd("..")
- puts "Running sbt doc from " + pwd + "; this may take a few minutes..."
- puts `sbt doc`
+ puts "Running sbt/sbt doc from " + pwd + "; this may take a few minutes..."
+ puts `sbt/sbt doc`
puts "Moving back into docs dir."
cd("docs")
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/api.md
----------------------------------------------------------------------
diff --git a/docs/api.md b/docs/api.md
index 11e2c15..e86d077 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -3,7 +3,7 @@ layout: global
title: Spark API documentation (Scaladoc)
---
-Here you can find links to the Scaladoc generated for the Spark sbt subprojects. If the following links don't work, try running `sbt doc` from the Spark project home directory.
+Here you can find links to the Scaladoc generated for the Spark sbt subprojects. If the following links don't work, try running `sbt/sbt doc` from the Spark project home directory.
- [Spark](api/core/index.html)
- [Spark Examples](api/examples/index.html)
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/hadoop-third-party-distributions.md
----------------------------------------------------------------------
diff --git a/docs/hadoop-third-party-distributions.md b/docs/hadoop-third-party-distributions.md
index 141d475..de6a2b0 100644
--- a/docs/hadoop-third-party-distributions.md
+++ b/docs/hadoop-third-party-distributions.md
@@ -12,7 +12,7 @@ with these distributions:
When compiling Spark, you'll need to
[set the SPARK_HADOOP_VERSION flag](index.html#a-note-about-hadoop-versions):
- SPARK_HADOOP_VERSION=1.0.4 sbt assembly
+ SPARK_HADOOP_VERSION=1.0.4 sbt/sbt assembly
The table below lists the corresponding `SPARK_HADOOP_VERSION` code for each CDH/HDP release. Note that
some Hadoop releases are binary compatible across client versions. This means the pre-built Spark
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index bf8d1c3..86d574d 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -17,7 +17,7 @@ Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). All you n
Spark uses [Simple Build Tool](http://www.scala-sbt.org), which is bundled with it. To compile the code, go into the top-level Spark directory and run
- sbt assembly
+ sbt/sbt assembly
For its Scala API, Spark {{site.SPARK_VERSION}} depends on Scala {{site.SCALA_VERSION}}. If you write applications in Scala, you will need to use this same version of Scala in your own program -- newer major versions may not work. You can get the right version of Scala from [scala-lang.org](http://www.scala-lang.org/download/).
@@ -56,12 +56,12 @@ Hadoop, you must build Spark against the same version that your cluster uses.
By default, Spark links to Hadoop 1.0.4. You can change this by setting the
`SPARK_HADOOP_VERSION` variable when compiling:
- SPARK_HADOOP_VERSION=2.2.0 sbt assembly
+ SPARK_HADOOP_VERSION=2.2.0 sbt/sbt assembly
In addition, if you wish to run Spark on [YARN](running-on-yarn.html), set
`SPARK_YARN` to `true`:
- SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt assembly
+ SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly
Note that on Windows, you need to set the environment variables on separate lines, e.g., `set SPARK_HADOOP_VERSION=1.2.1`.
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/python-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/python-programming-guide.md b/docs/python-programming-guide.md
index 5d48cb6..dc187b3 100644
--- a/docs/python-programming-guide.md
+++ b/docs/python-programming-guide.md
@@ -69,7 +69,7 @@ The script automatically adds the `bin/pyspark` package to the `PYTHONPATH`.
The `bin/pyspark` script launches a Python interpreter that is configured to run PySpark applications. To use `pyspark` interactively, first build Spark, then launch it directly from the command line without any options:
{% highlight bash %}
-$ sbt assembly
+$ sbt/sbt assembly
$ ./bin/pyspark
{% endhighlight %}
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/quick-start.md
----------------------------------------------------------------------
diff --git a/docs/quick-start.md b/docs/quick-start.md
index 9b9261c..153081b 100644
--- a/docs/quick-start.md
+++ b/docs/quick-start.md
@@ -12,7 +12,7 @@ See the [programming guide](scala-programming-guide.html) for a more complete re
To follow along with this guide, you only need to have successfully built Spark on one machine. Simply go into your Spark directory and run:
{% highlight bash %}
-$ sbt assembly
+$ sbt/sbt assembly
{% endhighlight %}
# Interactive Analysis with the Spark Shell
@@ -146,7 +146,7 @@ If you also wish to read data from Hadoop's HDFS, you will also need to add a de
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "<your-hdfs-version>"
{% endhighlight %}
-Finally, for sbt to work correctly, we'll need to layout `SimpleApp.scala` and `simple.sbt` according to the typical directory structure. Once that is in place, we can create a JAR package containing the application's code, then use `sbt run` to execute our program.
+Finally, for sbt to work correctly, we'll need to layout `SimpleApp.scala` and `simple.sbt` according to the typical directory structure. Once that is in place, we can create a JAR package containing the application's code, then use `sbt/sbt run` to execute our program.
{% highlight bash %}
$ find .
@@ -157,8 +157,8 @@ $ find .
./src/main/scala
./src/main/scala/SimpleApp.scala
-$ sbt package
-$ sbt run
+$ sbt/sbt package
+$ sbt/sbt run
...
Lines with a: 46, Lines with b: 23
{% endhighlight %}
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/running-on-yarn.md
----------------------------------------------------------------------
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index a35e003..717071d 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -12,7 +12,7 @@ was added to Spark in version 0.6.0, and improved in 0.7.0 and 0.8.0.
We need a consolidated Spark JAR (which bundles all the required dependencies) to run Spark jobs on a YARN cluster.
This can be built by setting the Hadoop version and `SPARK_YARN` environment variable, as follows:
- SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt assembly
+ SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly
The assembled JAR will be something like this:
`./assembly/target/scala-{{site.SCALA_VERSION}}/spark-assembly_{{site.SPARK_VERSION}}-hadoop2.0.5.jar`.
@@ -25,7 +25,7 @@ The build process now also supports new YARN versions (2.2.x). See below.
- The assembled jar can be installed into HDFS or used locally.
- Your application code must be packaged into a separate JAR file.
-If you want to test out the YARN deployment mode, you can use the current Spark examples. A `spark-examples_{{site.SCALA_VERSION}}-{{site.SPARK_VERSION}}` file can be generated by running `sbt assembly`. NOTE: since the documentation you're reading is for Spark version {{site.SPARK_VERSION}}, we are assuming here that you have downloaded Spark {{site.SPARK_VERSION}} or checked it out of source control. If you are using a different version of Spark, the version numbers in the jar generated by the sbt package command will obviously be different.
+If you want to test out the YARN deployment mode, you can use the current Spark examples. A `spark-examples_{{site.SCALA_VERSION}}-{{site.SPARK_VERSION}}` file can be generated by running `sbt/sbt assembly`. NOTE: since the documentation you're reading is for Spark version {{site.SPARK_VERSION}}, we are assuming here that you have downloaded Spark {{site.SPARK_VERSION}} or checked it out of source control. If you are using a different version of Spark, the version numbers in the jar generated by the sbt package command will obviously be different.
# Configuration
@@ -72,7 +72,7 @@ The command to launch the YARN Client is as follows:
For example:
# Build the Spark assembly JAR and the Spark examples JAR
- $ SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt assembly
+ $ SPARK_HADOOP_VERSION=2.0.5-alpha SPARK_YARN=true sbt/sbt assembly
# Configure logging
$ cp conf/log4j.properties.template conf/log4j.properties
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/docs/scala-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/scala-programming-guide.md b/docs/scala-programming-guide.md
index 3d0e892..c1ef46a 100644
--- a/docs/scala-programming-guide.md
+++ b/docs/scala-programming-guide.md
@@ -31,7 +31,7 @@ In addition, if you wish to access an HDFS cluster, you need to add a dependency
artifactId = hadoop-client
version = <your-hdfs-version>
-For other build systems, you can run `sbt assembly` to pack Spark and its dependencies into one JAR (`assembly/target/scala-{{site.SCALA_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop*.jar`), then add this to your CLASSPATH. Set the HDFS version as described [here](index.html#a-note-about-hadoop-versions).
+For other build systems, you can run `sbt/sbt assembly` to pack Spark and its dependencies into one JAR (`assembly/target/scala-{{site.SCALA_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop*.jar`), then add this to your CLASSPATH. Set the HDFS version as described [here](index.html#a-note-about-hadoop-versions).
Finally, you need to import some Spark classes and implicit conversions into your program. Add the following lines:
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/make-distribution.sh
----------------------------------------------------------------------
diff --git a/make-distribution.sh b/make-distribution.sh
index 6c466c8..61e6654 100755
--- a/make-distribution.sh
+++ b/make-distribution.sh
@@ -44,13 +44,16 @@ DISTDIR="$FWDIR/dist"
# Get version from SBT
export TERM=dumb # Prevents color codes in SBT output
-if ! test `which sbt` ;then
+VERSIONSTRING=$FWDIR/sbt/sbt "show version"
+
+if [ $? == -1 ] ;then
echo -e "You need sbt installed and available on your path."
echo -e "Download sbt from http://www.scala-sbt.org/"
exit -1;
fi
-VERSION=$(sbt "show version" | tail -1 | cut -f 2 | sed 's/^\([a-zA-Z0-9.-]*\).*/\1/')
+VERSION=$(echo "${VERSIONSTRING}" | tail -1 | cut -f 2 | sed 's/^\([a-zA-Z0-9.-]*\).*/\1/')
+echo "Version is ${VERSION}"
# Initialize defaults
SPARK_HADOOP_VERSION=1.0.4
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d86dc74d/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
index 6d2caca..09cc5a0 100755
--- a/sbt/sbt
+++ b/sbt/sbt
@@ -27,22 +27,17 @@ else
wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
else
printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
- exit
+ exit -1
fi
fi
if [ ! -f ${JAR} ]; then
# We failed to download
- printf "Our attempt to download sbt locally to {$JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
- exit
+ printf "Our attempt to download sbt locally to ${JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
+ exit -1
fi
printf "Launching sbt from .sbtlib\n"
java \
- -Duser.timezone=UTC \
- -Djava.awt.headless=true \
- -Dfile.encoding=UTF-8 \
- -XX:MaxPermSize=256m \
- -Xmx1g \
- -noverify \
+ -Xmx1200m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=256m \
-jar ${JAR} \
"$@"
fi
[10/14] git commit: Finish documentation changes
Posted by rx...@apache.org.
Finish documentation changes
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/7d0094bb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/7d0094bb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/7d0094bb
Branch: refs/heads/master
Commit: 7d0094bb562926af1d9b84a19d85329674aee003
Parents: 5a598b2
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sun Jan 5 22:12:47 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sun Jan 5 22:12:47 2014 -0800
----------------------------------------------------------------------
bin/pyspark | 2 +-
bin/spark-class | 2 +-
make-distribution.sh | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7d0094bb/bin/pyspark
----------------------------------------------------------------------
diff --git a/bin/pyspark b/bin/pyspark
index f97dfa7..d6810f4 100755
--- a/bin/pyspark
+++ b/bin/pyspark
@@ -31,7 +31,7 @@ if [ ! -f "$FWDIR/RELEASE" ]; then
ls "$FWDIR"/assembly/target/scala-$SCALA_VERSION/spark-assembly*hadoop*.jar >& /dev/null
if [[ $? != 0 ]]; then
echo "Failed to find Spark assembly in $FWDIR/assembly/target" >&2
- echo "You need to build Spark with sbt assembly before running this program" >&2
+ echo "You need to build Spark with sbt/sbt assembly before running this program" >&2
exit 1
fi
fi
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7d0094bb/bin/spark-class
----------------------------------------------------------------------
diff --git a/bin/spark-class b/bin/spark-class
index 49b0bef..c4225a3 100755
--- a/bin/spark-class
+++ b/bin/spark-class
@@ -104,7 +104,7 @@ if [ ! -f "$FWDIR/RELEASE" ]; then
jars_list=$(ls "$FWDIR"/assembly/target/scala-$SCALA_VERSION/ | grep "spark-assembly.*hadoop.*.jar")
if [ "$num_jars" -eq "0" ]; then
echo "Failed to find Spark assembly in $FWDIR/assembly/target/scala-$SCALA_VERSION/" >&2
- echo "You need to build Spark with 'sbt assembly' before running this program." >&2
+ echo "You need to build Spark with 'sbt/sbt assembly' before running this program." >&2
exit 1
fi
if [ "$num_jars" -gt "1" ]; then
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/7d0094bb/make-distribution.sh
----------------------------------------------------------------------
diff --git a/make-distribution.sh b/make-distribution.sh
index 61e6654..1a3a5d0 100755
--- a/make-distribution.sh
+++ b/make-distribution.sh
@@ -95,7 +95,7 @@ export SPARK_HADOOP_VERSION
export SPARK_YARN
cd $FWDIR
-"sbt" "assembly/assembly"
+"sbt/sbt" "assembly/assembly"
# Make directories
rm -rf "$DISTDIR"
[02/14] git commit: Pass commands down to system sbt as well
Posted by rx...@apache.org.
Pass commands down to system sbt as well
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/97123be1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/97123be1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/97123be1
Branch: refs/heads/master
Commit: 97123be1d7d1b68ec0cda09fd5894fc4af5f82c5
Parents: 9e9a913
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sat Jan 4 20:16:56 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sat Jan 4 20:16:56 2014 -0800
----------------------------------------------------------------------
sbt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/97123be1/sbt
----------------------------------------------------------------------
diff --git a/sbt b/sbt
index e04a20c..8cff6df 100755
--- a/sbt
+++ b/sbt
@@ -11,7 +11,7 @@ printf "Checking for system sbt ["
if hash sbt 2>/dev/null; then
printf "FOUND]\n"
# Use System SBT
- sbt
+ sbt $@
else
printf "NOT FOUND]\n"
# Download sbt or use allready downloaded
[09/14] git commit: Fix indentatation
Posted by rx...@apache.org.
Fix indentatation
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/5a598b2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/5a598b2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/5a598b2d
Branch: refs/heads/master
Commit: 5a598b2d7b72db7f732a63e6e92d54786f68ee1e
Parents: d86dc74
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sun Jan 5 22:07:32 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sun Jan 5 22:07:32 2014 -0800
----------------------------------------------------------------------
sbt/sbt | 32 ++++++++++++++++----------------
1 file changed, 16 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/5a598b2d/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
index 09cc5a0..0951e9d 100755
--- a/sbt/sbt
+++ b/sbt/sbt
@@ -16,28 +16,28 @@ else
printf "NOT FOUND]\n"
# Download sbt or use already downloaded
if [ ! -d .sbtlib ]; then
- mkdir .sbtlib
+ mkdir .sbtlib
fi
if [ ! -f ${JAR} ]; then
- # Download
- printf "Attempting to fetch sbt\n"
- if hash curl 2>/dev/null; then
- curl --progress-bar ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}
- elif hash wget 2>/dev/null; then
- wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
- else
- printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
- exit -1
- fi
+ # Download
+ printf "Attempting to fetch sbt\n"
+ if hash curl 2>/dev/null; then
+ curl --progress-bar ${URL1} > ${JAR} || curl --progress-bar ${URL2} > ${JAR}
+ elif hash wget 2>/dev/null; then
+ wget --progress=bar ${URL1} -O ${JAR} || wget --progress=bar ${URL2} -O ${JAR}
+ else
+ printf "You do not have curl or wget installed, please install sbt manually from http://www.scala-sbt.org/\n"
+ exit -1
+ fi
fi
if [ ! -f ${JAR} ]; then
- # We failed to download
- printf "Our attempt to download sbt locally to ${JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
- exit -1
+ # We failed to download
+ printf "Our attempt to download sbt locally to ${JAR} failed. Please install sbt manually from http://www.scala-sbt.org/\n"
+ exit -1
fi
printf "Launching sbt from .sbtlib\n"
java \
-Xmx1200m -XX:MaxPermSize=350m -XX:ReservedCodeCacheSize=256m \
- -jar ${JAR} \
- "$@"
+ -jar ${JAR} \
+ "$@"
fi
[12/14] git commit: Put quote arround arguments passed down to system
sbt
Posted by rx...@apache.org.
Put quote arround arguments passed down to system sbt
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b590adb2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b590adb2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b590adb2
Branch: refs/heads/master
Commit: b590adb2ad06fbb5f38c55aa16369c878d84839e
Parents: 2dc83de
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Mon Jan 6 23:31:39 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Mon Jan 6 23:31:39 2014 -0800
----------------------------------------------------------------------
sbt/sbt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b590adb2/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
index af422d9..2c19987 100755
--- a/sbt/sbt
+++ b/sbt/sbt
@@ -11,7 +11,7 @@ printf "Checking for system sbt ["
if hash sbt 2>/dev/null; then
printf "FOUND]\n"
# Use System SBT
- sbt $@
+ sbt "$@"
else
printf "NOT FOUND]\n"
# Download sbt or use already downloaded
[04/14] git commit: Spelling
Posted by rx...@apache.org.
Spelling
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d2a5c75a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d2a5c75a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/d2a5c75a
Branch: refs/heads/master
Commit: d2a5c75a4d3ca5123fb77ad24beb1b4474b36a4f
Parents: b4a1ffc
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sat Jan 4 21:44:04 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sat Jan 4 21:44:04 2014 -0800
----------------------------------------------------------------------
sbt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d2a5c75a/sbt
----------------------------------------------------------------------
diff --git a/sbt b/sbt
index 8cff6df..d21806e 100755
--- a/sbt
+++ b/sbt
@@ -14,7 +14,7 @@ if hash sbt 2>/dev/null; then
sbt $@
else
printf "NOT FOUND]\n"
- # Download sbt or use allready downloaded
+ # Download sbt or use already downloaded
if [ ! -d .sbtlib ]; then
mkdir .sbtlib
fi
[06/14] git commit: And update docs to match
Posted by rx...@apache.org.
And update docs to match
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/d7d95a09
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/d7d95a09
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/d7d95a09
Branch: refs/heads/master
Commit: d7d95a099f5f215aff465f948446a8b68113b391
Parents: 0d6700e
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sat Jan 4 21:45:22 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sat Jan 4 21:45:22 2014 -0800
----------------------------------------------------------------------
.gitignore | 2 +-
README.md | 4 ++--
sbt/sbt | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d7d95a09/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index 1692bde..39635d7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,7 +4,7 @@
*.iml
*.iws
.idea/
-.sbtlib/*.jar
+sbt/*.jar
.settings
.cache
/build/
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d7d95a09/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index db1e2c4..2c08a4a 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ This README file only contains basic setup instructions.
Spark requires Scala 2.10. The project is built using Simple Build Tool (SBT),
which can be obtained [here](http://www.scala-sbt.org). To build Spark and its example programs, run:
- ./sbt assembly
+ ./sbt/sbt assembly
Once you've built Spark, the easiest way to start using it is the shell:
@@ -41,7 +41,7 @@ locally with one thread, or "local[N]" to run locally with N threads.
Testing first requires [Building](#Building) Spark. Once Spark is built, tests
can be run using:
-`./sbt test`
+`./sbt/sbt test`
## A Note About Hadoop Versions
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/d7d95a09/sbt/sbt
----------------------------------------------------------------------
diff --git a/sbt/sbt b/sbt/sbt
index d21806e..a7146e3 100755
--- a/sbt/sbt
+++ b/sbt/sbt
@@ -5,7 +5,7 @@
SBT_VERSION=0.12.4
URL1=http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
URL2=http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
-JAR=.sbtlib/sbt-launch-${SBT_VERSION}.jar
+JAR=sbt/sbt-launch-${SBT_VERSION}.jar
printf "Checking for system sbt ["
if hash sbt 2>/dev/null; then
[03/14] git commit: Switch from sbt to ./sbt in the README file
Posted by rx...@apache.org.
Switch from sbt to ./sbt in the README file
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b4a1ffc6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b4a1ffc6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b4a1ffc6
Branch: refs/heads/master
Commit: b4a1ffc6c2634118bb1d07216221b862c32d6397
Parents: 97123be
Author: Holden Karau <ho...@pigscanfly.ca>
Authored: Sat Jan 4 20:17:30 2014 -0800
Committer: Holden Karau <ho...@pigscanfly.ca>
Committed: Sat Jan 4 20:17:30 2014 -0800
----------------------------------------------------------------------
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b4a1ffc6/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 6daa463..db1e2c4 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ This README file only contains basic setup instructions.
Spark requires Scala 2.10. The project is built using Simple Build Tool (SBT),
which can be obtained [here](http://www.scala-sbt.org). To build Spark and its example programs, run:
- sbt assembly
+ ./sbt assembly
Once you've built Spark, the easiest way to start using it is the shell:
@@ -41,7 +41,7 @@ locally with one thread, or "local[N]" to run locally with N threads.
Testing first requires [Building](#Building) Spark. Once Spark is built, tests
can be run using:
-`sbt test`
+`./sbt test`
## A Note About Hadoop Versions
[14/14] git commit: Merge pull request #331 from holdenk/master
Posted by rx...@apache.org.
Merge pull request #331 from holdenk/master
Add a script to download sbt if not present on the system
As per the discussion on the dev mailing list this script will use the system sbt if present or otherwise attempt to install the sbt launcher. The fall back error message in the event it fails instructs the user to install sbt. While the URLs it fetches from aren't controlled by the spark project directly, they are stable and the current authoritative sources.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/a862cafa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/a862cafa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/a862cafa
Branch: refs/heads/master
Commit: a862cafacf555373b5fdbafb4c9c4d712b191648
Parents: b97ef21 60a7a6b
Author: Reynold Xin <rx...@apache.org>
Authored: Tue Jan 7 00:18:20 2014 -0800
Committer: Reynold Xin <rx...@apache.org>
Committed: Tue Jan 7 00:18:20 2014 -0800
----------------------------------------------------------------------
.gitignore | 1 +
README.md | 18 ++++++-----
bin/pyspark | 2 +-
bin/run-example | 2 +-
bin/spark-class | 2 +-
docs/README.md | 4 +--
docs/_plugins/copy_api_dirs.rb | 4 +--
docs/api.md | 2 +-
docs/hadoop-third-party-distributions.md | 2 +-
docs/index.md | 6 ++--
docs/python-programming-guide.md | 2 +-
docs/quick-start.md | 8 ++---
docs/running-on-yarn.md | 6 ++--
docs/scala-programming-guide.md | 2 +-
make-distribution.sh | 9 ++++--
project/build.properties | 1 -
sbt/sbt | 43 +++++++++++++++++++++++++++
17 files changed, 81 insertions(+), 33 deletions(-)
----------------------------------------------------------------------