You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by napsternxg <gi...@git.apache.org> on 2016/01/19 21:26:38 UTC
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
GitHub user napsternxg opened a pull request:
https://github.com/apache/spark/pull/10836
Fixes SPARK-12910: R version for installing sparkR
Testing code:
```
$ ./install-dev.sh
USING R_HOME = /usr/bin
ERROR: this R is version 2.15.1, package 'SparkR' requires R >= 3.0
```
Using the new argument:
```
$ ./install-dev.sh /content/username/SOFTWARE/R-3.2.3
USING R_HOME = /content/username/SOFTWARE/R-3.2.3/bin
* installing *source* package ‘SparkR’ ...
** R
** inst
** preparing package for lazy loading
Creating a new generic function for ‘colnames’ in package ‘SparkR’
Creating a new generic function for ‘colnames<-’ in package ‘SparkR’
Creating a new generic function for ‘cov’ in package ‘SparkR’
Creating a new generic function for ‘na.omit’ in package ‘SparkR’
Creating a new generic function for ‘filter’ in package ‘SparkR’
Creating a new generic function for ‘intersect’ in package ‘SparkR’
Creating a new generic function for ‘sample’ in package ‘SparkR’
Creating a new generic function for ‘transform’ in package ‘SparkR’
Creating a new generic function for ‘subset’ in package ‘SparkR’
Creating a new generic function for ‘summary’ in package ‘SparkR’
Creating a new generic function for ‘lag’ in package ‘SparkR’
Creating a new generic function for ‘rank’ in package ‘SparkR’
Creating a new generic function for ‘sd’ in package ‘SparkR’
Creating a new generic function for ‘var’ in package ‘SparkR’
Creating a new generic function for ‘predict’ in package ‘SparkR’
Creating a new generic function for ‘rbind’ in package ‘SparkR’
Creating a generic function for ‘lapply’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘Filter’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘alias’ from package ‘stats’ in package ‘SparkR’
Creating a generic function for ‘substr’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘%in%’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘mean’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘unique’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘nrow’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘ncol’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘head’ from package ‘utils’ in package ‘SparkR’
Creating a generic function for ‘factorial’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘atan2’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘ifelse’ from package ‘base’ in package ‘SparkR’
** help
No man pages found in package ‘SparkR’
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (SparkR)
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/napsternxg/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10836.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10836
----
commit 9a41d47dfa5b778744972163a53b2324a2bc54a2
Author: Shubhanshu Mishra <sm...@illinois.edu>
Date: 2016-01-19T20:13:28Z
Fixes SPARK-12910: R version for installing sparkR
I have added an option for passing the R_HOME path to the
`install-dev.sh` file which fixes SPARK-12910.
The code works and has been tested.
commit 17fe9161a9655dcbe609ec86f0770a100eb9387d
Author: Shubhanshu Mishra <sm...@illinois.edu>
Date: 2016-01-19T20:15:59Z
Merge branch 'master' of github.com:apache/spark
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50181745
--- Diff: R/install-dev.sh ---
@@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
mkdir -p $LIB_DIR
pushd $FWDIR > /dev/null
+if [ ! -z "$1" ]
+ then
+ R_HOME="$1/bin"
+ else
+ R_HOME="$(dirname $(which R))"
--- End diff --
Can we just use a `R_HOME` if it is set instead of using a command line argument ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173421200
looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50190846
--- Diff: R/install-dev.sh ---
@@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
mkdir -p $LIB_DIR
pushd $FWDIR > /dev/null
+if [ ! -z "$1" ]
+ then
+ R_HOME="$1/bin"
+ else
+ R_HOME="$(dirname $(which R))"
--- End diff --
Updated my pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173427668
Merging this to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50215117
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
I think this is needed as mentioned in running sparkR from R studio. The
script there tries to access the lib location which might not be present by
default in the sparkR folder if the wrong version of R is selected by
default.
*Regards,*
*Shubhanshu Mishra*
Research Assistant,
iSchool at University of Illinois at Urbana-Champaign
--------------------------------------------------
*Website:* http://shubhanshu.com
*LinkedIn Profile: *http://www.linkedin.com/in/shubhanshumishra
Blog <http://smexyyweby.wordpress.com/> || Facebook
<http://www.facebook.com/shubhanshu.mishra> || Twitter
<http://www.twitter.com/TheShubhanshu> || LinkedIn
<http://www.linkedin.com/in/shubhanshumishra>
On Tue, Jan 19, 2016 at 10:03 PM, Felix Cheung <no...@github.com>
wrote:
> In R/README.md
> <https://github.com/apache/spark/pull/10836#discussion_r50212459>:
>
> > @@ -1,6 +1,16 @@
> > # R on Spark
> >
> > SparkR is an R package that provides a light-weight frontend to use Spark from R.
> > +### Installing sparkR
> > +
> > +Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
>
> To clarify, I'm referring the part about "sparkR need to be created in
> $SPARK_HOME/R/lib. This can be done by running the script
> $SPARK_HOME/R/install-dev.sh."
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/10836/files#r50212459>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50189970
--- Diff: R/install-dev.sh ---
@@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
mkdir -p $LIB_DIR
pushd $FWDIR > /dev/null
+if [ ! -z "$1" ]
+ then
+ R_HOME="$1/bin"
+ else
+ R_HOME="$(dirname $(which R))"
--- End diff --
I have made the fix.
Here are my new test results:
```
$ unset R_HOME
ram:/content/username/SOFTWARE/spark/R $ ./install-dev.sh
USING R_HOME =
ERROR: this R is version 2.15.1, package 'SparkR' requires R >= 3.0
* removing ‘/content/username/SOFTWARE/spark/R/lib/SparkR’
ram:/content/username/SOFTWARE/spark/R $ export R_HOME=/content/username/SOFTWARE/R-3.2.3
ram:/content/username/SOFTWARE/spark/R $ ./install-dev.sh
USING R_HOME = /content/username/SOFTWARE/R-3.2.3
* installing *source* package ‘SparkR’ ...
** R
** inst
** preparing package for lazy loading
Creating a new generic function for ‘colnames’ in package ‘SparkR’
Creating a new generic function for ‘colnames<-’ in package ‘SparkR’
Creating a new generic function for ‘cov’ in package ‘SparkR’
Creating a new generic function for ‘na.omit’ in package ‘SparkR’
Creating a new generic function for ‘filter’ in package ‘SparkR’
Creating a new generic function for ‘intersect’ in package ‘SparkR’
Creating a new generic function for ‘sample’ in package ‘SparkR’
Creating a new generic function for ‘transform’ in package ‘SparkR’
Creating a new generic function for ‘subset’ in package ‘SparkR’
Creating a new generic function for ‘summary’ in package ‘SparkR’
Creating a new generic function for ‘lag’ in package ‘SparkR’
Creating a new generic function for ‘rank’ in package ‘SparkR’
Creating a new generic function for ‘sd’ in package ‘SparkR’
Creating a new generic function for ‘var’ in package ‘SparkR’
Creating a new generic function for ‘predict’ in package ‘SparkR’
Creating a new generic function for ‘rbind’ in package ‘SparkR’
Creating a generic function for ‘lapply’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘Filter’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘alias’ from package ‘stats’ in package ‘SparkR’
Creating a generic function for ‘substr’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘%in%’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘mean’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘unique’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘nrow’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘ncol’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘head’ from package ‘utils’ in package ‘SparkR’
Creating a generic function for ‘factorial’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘atan2’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘ifelse’ from package ‘base’ in package ‘SparkR’
** help
No man pages found in package ‘SparkR’
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (SparkR)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50184164
--- Diff: R/install-dev.sh ---
@@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
mkdir -p $LIB_DIR
pushd $FWDIR > /dev/null
+if [ ! -z "$1" ]
+ then
+ R_HOME="$1/bin"
+ else
+ R_HOME="$(dirname $(which R))"
--- End diff --
In R there there is usually no `R_HOME` like there is in python so this might confuse users. We should use the default R when the variable is not passed.
Or I can use a variable like `SPARK_R_HOME` which specifies the R home directory ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-172975965
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173000126
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173019976
**[Test build #49721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49721/consoleFull)** for PR 10836 at commit [`0b3960a`](https://github.com/apache/spark/commit/0b3960a29dd47ae12c9534806191cfb098139888).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50206183
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
If one were running from R studio with the steps in `Using SparkR from RStudio` below he wouldn't have to install or run install-dev.sh though - could we clarify that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173004868
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50229721
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
Great. So is this PR ready for a merge into the master ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50207105
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
I am running this on a machine with no X-server hence no R-studio, so this kind of functionality will be needed for users like me.
Even for R-studio I feel sparkR needs to be compiled with the version of R >= 3.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173004870
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49709/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50225325
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
ok I think I get your point now.
I guess we are saying this README.md is more for developer, so I'm ok with what you have here.
There are users that are not building Spark from source and are running with the binary release, in which case the SPARK_HOME/R/lib is there and they would not need to install the SparkR package. Similarly when running SparkR with a cluster manager, on the worker nodes SparkR would not need to be installed either. I agree they are possibly outside the scope of this file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50215136
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
I think this is needed as mentioned in running sparkR from R studio. The script there tries to access the lib location which might not be present by default in the sparkR folder if the wrong version of R is selected by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50212459
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
To clarify, I'm referring the part about "sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50188851
--- Diff: R/install-dev.sh ---
@@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
mkdir -p $LIB_DIR
pushd $FWDIR > /dev/null
+if [ ! -z "$1" ]
+ then
+ R_HOME="$1/bin"
+ else
+ R_HOME="$(dirname $(which R))"
--- End diff --
`R_HOME` is a reasonably well defined concept ? http://www.inside-r.org/r-doc/base/R.home
Regardless I think you will need to use a different R binary for running sparkR if you don't want it to use the default R ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173023865
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173420721
looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10836
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173035589
LGTM. cc @felixcheung
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50184402
--- Diff: R/install-dev.sh ---
@@ -35,12 +35,19 @@ LIB_DIR="$FWDIR/lib"
mkdir -p $LIB_DIR
pushd $FWDIR > /dev/null
+if [ ! -z "$1" ]
+ then
+ R_HOME="$1/bin"
+ else
+ R_HOME="$(dirname $(which R))"
--- End diff --
Also if using an environment variable then the code for running sparkR would also have to be edited to use the R version from this path.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173023868
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49721/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by napsternxg <gi...@git.apache.org>.
Github user napsternxg commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50224572
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
I am sorry if my I was not clear before but what I mean is the following:
1. In order for the code `.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))` to work, the directory `$SPARK_HOME/R/lib` needs to exist. When we build spark using `build/mvn -DskipTests clean package` this directory is not created by default. Hence we have to run `install-dev.sh` in order to use SparkR from an R shell.
Also, if we look at the code in `install-dev.sh`, the following lines actually create the `lib` directory.
```
FWDIR="$(cd `dirname $0`; pwd)"
LIB_DIR="$FWDIR/lib"
mkdir -p $LIB_DIR
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/10836#discussion_r50219109
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
# R on Spark
SparkR is an R package that provides a light-weight frontend to use Spark from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --
actually, this line
```.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))```
added SPARK_HOME/R/lib into R's lib path and allows R, any running version, to load SparkR package from there - SparkR packages does not need to be installed with `R CMD INSTALL` (in install-dev.sh) at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10836#issuecomment-173023722
**[Test build #49721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49721/consoleFull)** for PR 10836 at commit [`0b3960a`](https://github.com/apache/spark/commit/0b3960a29dd47ae12c9534806191cfb098139888).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org