You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nicklan <gi...@git.apache.org> on 2014/03/27 19:07:28 UTC
[GitHub] spark pull request: Make sed do -i '' on OSX
GitHub user nicklan opened a pull request:
https://github.com/apache/spark/pull/258
Make sed do -i '' on OSX
I don't have access to an OSX machine, so if someone could test this that would be great.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nicklan/spark osx-sed-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/258.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #258
----
commit 558fd6eaabf35a12bda6f3305aa4eefd51183d01
Author: Nick Lanham <ni...@afternight.org>
Date: 2014-03-27T18:06:13Z
Make sed do -i '' on OSX
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38846370
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13523/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38846369
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38880262
Oh one other thing, make an account on our JIRA and I can assign https://spark-project.atlassian.net/browse/SPARK-1326 to you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38842503
Merged build triggered. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38880156
Actually nm, saw you fixed it. This fixes make-distribution.sh, although unfortunately Tachyon as packaged here doesn't properly deploy on OS X. If I go into `dist`, add a `conf/slaves` file, and run `sbin/start-all.sh --with-tachyon`, I get:
```
starting org.apache.spark.deploy.master.Master, logging to /Users/matei/workspace/apache-spark/dist/sbin/../logs/spark-matei-org.apache.spark.deploy.master.Master-1-mbp-3.out
sed: 1: "/Users/matei/workspace/ ...": invalid command code m
awk: can't open file /proc/meminfo
source line number 1
sed: 1: "/Users/matei/workspace/ ...": invalid command code m
localhost: Formatting Tachyon Worker @ mbp-3
localhost: Error: Could not find or load main class tachyon.Format
Formatting Tachyon Master @ localhost
Error: Could not find or load main class tachyon.Format
```
However we can fix this in another JIRA, so I'll merge this PR as is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38843162
@nicklan I actually see some other errors due to pushd and popd now:
```
./make-distribution.sh: line 133: pushd: no other directory
Fetchting tachyon tgz
--2014-03-27 11:31:11-- https://github.com/amplab/tachyon/releases/download/v0.4.1/tachyon-0.4.1-bin.tar.gz
Resolving github.com... 192.30.252.131
Connecting to github.com|192.30.252.131|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/github-cloud/releases/7276954/dedb4c68-a574-11e3-960d-341edab9bc3c.gz?response-content-disposition=attachment%3B%20filename%3Dtachyon-0.4.1-bin.tar.gz&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1395945132&Signature=3%2FZTB8tWWf89B8pc2duZ6gl6n%2F4%3D [following]
--2014-03-27 11:31:12-- https://s3.amazonaws.com/github-cloud/releases/7276954/dedb4c68-a574-11e3-960d-341edab9bc3c.gz?response-content-disposition=attachment%3B%20filename%3Dtachyon-0.4.1-bin.tar.gz&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1395945132&Signature=3%2FZTB8tWWf89B8pc2duZ6gl6n%2F4%3D
Resolving s3.amazonaws.com... 207.171.189.80
Connecting to s3.amazonaws.com|207.171.189.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22751619 (22M) [application/octet-stream]
Saving to: 'tachyon-0.4.1-bin.tar.gz'
100%[======================================>] 22,751,619 3.79MB/s in 9.5s
2014-03-27 11:31:23 (2.28 MB/s) - 'tachyon-0.4.1-bin.tar.gz' saved [22751619/22751619]
./make-distribution.sh: line 151: popd: directory stack empty
```
Haven't looked into it much, see if you can get someone with a Mac to work with.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by nicklan <gi...@git.apache.org>.
Github user nicklan commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38880675
Okay will do, and I can look at these other errors tomorrow
On Mar 27, 2014 6:20 PM, "Matei Zaharia" <no...@github.com> wrote:
> Oh one other thing, make an account on our JIRA and I can assign
> https://spark-project.atlassian.net/browse/SPARK-1326 to you.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/258#issuecomment-38880262>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by nicklan <gi...@git.apache.org>.
Github user nicklan commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38846955
@mateiz okay, I think i figured out the mktemp problem. if you could test it once more that would be great. i'll also try and test it on a mac later today if you're too busy at the moment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38880181
(BTW if I run this twice, the "sed" error goes away but I still get "Error: Could not find or load main class tachyon.Format").
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38850768
Merged build finished. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38890491
Thanks, I've now merged this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38842086
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38847298
Merged build started. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/258
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38850878
Merged build finished. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by nicklan <gi...@git.apache.org>.
Github user nicklan commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38948413
Some of this is unfortunately a bug in the tachyon scripts. I'll submit a
PR there to get them fixed. Not sure it not being able to find the main
class however, that seems bad. I'll dig a bit.
On Thu, Mar 27, 2014 at 6:17 PM, Matei Zaharia <no...@github.com>wrote:
> Actually nm, saw you fixed it. This fixes make-distribution.sh, although
> unfortunately Tachyon as packaged here doesn't properly deploy on OS X. If
> I go into dist, add a conf/slaves file, and run sbin/start-all.sh
> --with-tachyon, I get:
>
> starting org.apache.spark.deploy.master.Master, logging to /Users/matei/workspace/apache-spark/dist/sbin/../logs/spark-matei-org.apache.spark.deploy.master.Master-1-mbp-3.out
> sed: 1: "/Users/matei/workspace/ ...": invalid command code m
> awk: can't open file /proc/meminfo
> source line number 1
> sed: 1: "/Users/matei/workspace/ ...": invalid command code m
> localhost: Formatting Tachyon Worker @ mbp-3
> localhost: Error: Could not find or load main class tachyon.Format
> Formatting Tachyon Master @ localhost
> Error: Could not find or load main class tachyon.Format
>
> However we can fix this in another JIRA, so I'll merge this PR as is.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/258#issuecomment-38880156>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38840027
Merged build started. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by nicklan <gi...@git.apache.org>.
Github user nicklan commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38844444
@mateiz huh, that's odd, it suggests that `mktemp -d` returned an empty string. Were there any other errors right before the pushd one? I also checked and that should work on OSX. I really have no one around with a Mac today, but I'll try and get this sorted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38879651
So it seems that on OS X, mktemp needs a template: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html. Just doing `mktemp -d` would fail. The best way to run it there will be `mktemp -d -t spark`, which will put a file in TMPDIR with "spark" in its name. If that also works on GNU mktemp, we can go wit that.
The pushd/popd errors are because mktemp failed, but that was earlier in the log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38850879
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13526/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by nicklan <gi...@git.apache.org>.
Github user nicklan commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38879709
Yeah, have you tried with my latests PR where I added the -t?
On Thu, Mar 27, 2014 at 6:08 PM, Matei Zaharia <no...@github.com>wrote:
> So it seems that on OS X, mktemp needs a template:
> https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html.
> Just doing mktemp -d would fail. The best way to run it there will be mktemp
> -d -t spark, which will put a file in TMPDIR with "spark" in its name. If
> that also works on GNU mktemp, we can go wit that.
>
> The pushd/popd errors are because mktemp failed, but that was earlier in
> the log.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/apache/spark/pull/258#issuecomment-38879651>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38850770
One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13525/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38847277
Merged build triggered. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38844395
Merged build started. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Make sed do -i '' on OSX
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/258#issuecomment-38840003
Merged build triggered. One or more automated tests failed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---