You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Adrian Bonar <Ad...@microsoft.com> on 2013/12/04 23:31:00 UTC
Pre-build Spark for Windows 8.1
Separate from my previous thread about building a distribution of Spark on Win8, I am also trying to run the pre-build binaries with little success. I downloaded and extract "spark-0.8.0-incubating-bin-hadoop1" to d:\spark and attempted to start a master with the following error:
$ sh bin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
failed to launch org.apache.spark.deploy.master.Master:
Error: Could not find or load main class org.apache.spark.deploy.master.Master
full log in /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
--Adrian (again. :))
RE: Pre-build Spark for Windows 8.1
Posted by Adrian Bonar <Ad...@microsoft.com>.
Strange, but that definitely did the trick. Thanks again!
From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
Sent: Thursday, December 5, 2013 2:44 PM
To: user@spark.incubator.apache.org
Subject: Re: Pre-build Spark for Windows 8.1
Hi,
When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the URL (uppercase instead of lowercase). Unfortunately Akka is very specific about seeing hostnames written in the same way on each node, or else it thinks the message is for another machine!
Matei
On Dec 5, 2013, at 8:27 AM, Adrian Bonar <Ad...@microsoft.com>> wrote:
The master starts up now as expected but the workers are unable to connect to the master. It looks like the master is refusing the connection messages but I'm not sure why. The first two error lines below are from trying to connect a worker from a separate machine and the last two error lines are from trying to connect a worker on the same machine as the master. I verified that the workers do not show up in the master's web ui.
MASTER:
D:\spark>spark-class org.apache.spark.deploy.master.Master
13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/05 08:08:34 INFO master.Master: Starting Spark master at spark://ADRIBONA-DEV-1:7077
13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/master/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
13/12/05 08:08:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0<ma...@0.0.0.0>:8088
13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at http://ADRIBONA-DEV-1:8088<http://adribona-dev-1:8088/>
13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master<akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker%5d,Actor%5bakka:/sparkMaster@adribona-dev-1:7077/user/Master>]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master atakka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master<akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker%5d,Actor%5bakka:/sparkMaster@adribona-dev-1:7077/user/Master>]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
WORKER:
D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker spark://adribona-dev-1:7077
13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/json,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/log,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/logPage,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
13/12/05 08:18:46 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0<ma...@0.0.0.0>:8081
13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at http://ADRIBONA-DEV-1:8081<http://adribona-dev-1:8081/>
13/12/05 08:18:46 INFO worker.Worker: Connecting to master spark://adribona-dev-1:7077
--Adrian
From: Adrian Bonar [mailto:Adrian.Bonar@microsoft.com]
Sent: Thursday, December 5, 2013 7:49 AM
To: user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>
Subject: RE: Pre-build Spark for Windows 8.1
Excellent! Thank you, Matei.
From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
Sent: Wednesday, December 4, 2013 4:26 PM
To: user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>
Subject: Re: Pre-build Spark for Windows 8.1
Hey Adrian,
Ideally you shouldn't use Cygwin to run on Windows - use the .cmd scripts we provide instead. Cygwin might be made to work but we haven't tried to do this so far so it's not supported. If you can fix it, that would of course be welcome.
Also, the deploy scripts don't work on Windows - we assumed that people would mostly use Windows for local development and testing. However, if you do want to launch a cluster, you can use spark-class.cmd manually to launch a Master and then Workers. Use the commands in the first part of http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
Matei
On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com>> wrote:
Separate from my previous thread about building a distribution of Spark on Win8, I am also trying to run the pre-build binaries with little success. I downloaded and extract "spark-0.8.0-incubating-bin-hadoop1" to d:\spark and attempted to start a master with the following error:
$ sh bin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
failed to launch org.apache.spark.deploy.master.Master:
Error: Could not find or load main class org.apache.spark.deploy.master.Master
full log in /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
--Adrian (again. :))
Re: Pre-build Spark for Windows 8.1
Posted by Andrew Ash <an...@andrewash.com>.
Thanks for the info Matei. Seems like lots of other users of Akka have
similar issues -- maybe at some point in the future it'll be worth making
this a bit more flexible, but there are more important places to spend time
right now.
On Fri, Dec 6, 2013 at 12:06 PM, Matei Zaharia <ma...@gmail.com>wrote:
> Hey Andrew, unfortunately I don’t know how easy this is. Maybe future
> versions of Akka have it. We can certainly ask them to do it in general but
> I imagine there are some use cases where they wanted this behavior.
>
> Matei
>
>
> On Dec 5, 2013, at 2:49 PM, Andrew Ash <an...@andrewash.com> wrote:
>
> Speaking of akka and host sensitivity... How much have you hacked on akka
> to get it to support all of: myhost.mydomain.int, myhost, and 10.1.1.1?
> It's kind of a pain to get the Spark URL to exactly match. I'm wondering
> if there are usability gains that could be made here or if we're pretty
> stuck.
>
>
> On Thu, Dec 5, 2013 at 2:43 PM, Matei Zaharia <ma...@gmail.com>wrote:
>
>> Hi,
>>
>> When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the
>> URL (uppercase instead of lowercase). Unfortunately Akka is very specific
>> about seeing hostnames written in the same way on each node, or else it
>> thinks the message is for another machine!
>>
>> Matei
>>
>> On Dec 5, 2013, at 8:27 AM, Adrian Bonar <Ad...@microsoft.com>
>> wrote:
>>
>> The master starts up now as expected but the workers are unable to
>> connect to the master. It looks like the master is refusing the connection
>> messages but I’m not sure why. The first two error lines below are from
>> trying to connect a worker from a separate machine and the last two error
>> lines are from trying to connect a worker on the same machine as the
>> master. I verified that the workers do not show up in the master’s web ui.
>>
>> MASTER:
>> D:\spark>spark-class org.apache.spark.deploy.master.Master
>> 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:08:34 INFO master.Master: Starting Spark master at
>> spark://ADRIBONA-DEV-1:7077
>> 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/master/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/app/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/app,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:08:34 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:8088
>> 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at
>> http://ADRIBONA-DEV-1:8088 <http://adribona-dev-1:8088/>
>> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message
>> RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1)
>> for non-local recipient
>> akka://sparkMaster@adribona-dev-1:7077/user/Master at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message
>> DaemonMsgWatch(Actor[
>> akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message
>> RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1)
>> for non-local recipient
>> akka://sparkMaster@adribona-dev-1:7077/user/Master at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message
>> DaemonMsgWatch(Actor[
>> akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>>
>> WORKER:
>> D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker
>> spark://adribona-dev-1:7077
>> 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker
>> ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
>> 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
>> 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/log,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/logPage,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:18:46 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:8081
>> 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at
>> http://ADRIBONA-DEV-1:8081 <http://adribona-dev-1:8081/>
>> 13/12/05 08:18:46 INFO worker.Worker: Connecting to master
>> spark://adribona-dev-1:7077
>>
>>
>> --Adrian
>>
>> *From:* Adrian Bonar [mailto:Adrian.Bonar@microsoft.com<Ad...@microsoft.com>
>> ]
>> *Sent:* Thursday, December 5, 2013 7:49 AM
>> *To:* user@spark.incubator.apache.org
>> *Subject:* RE: Pre-build Spark for Windows 8.1
>>
>> Excellent! Thank you, Matei.
>>
>> *From:* Matei Zaharia [mailto:matei.zaharia@gmail.com<ma...@gmail.com>
>> ]
>> *Sent:* Wednesday, December 4, 2013 4:26 PM
>> *To:* user@spark.incubator.apache.org
>> *Subject:* Re: Pre-build Spark for Windows 8.1
>>
>> Hey Adrian,
>>
>> Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts
>> we provide instead. Cygwin might be made to work but we haven’t tried to do
>> this so far so it’s not supported. If you can fix it, that would of course
>> be welcome.
>>
>> Also, the deploy scripts don’t work on Windows — we assumed that people
>> would mostly use Windows for local development and testing. However, if you
>> do want to launch a cluster, you can use spark-class.cmd manually to launch
>> a Master and then Workers. Use the commands in the first part of
>> http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
>>
>> Matei
>>
>> On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com>
>> wrote:
>>
>>
>>
>> Separate from my previous thread about *building* a distribution of
>> Spark on Win8, I am also trying to run the pre-build binaries with little
>> success. I downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to
>> d:\spark and attempted to start a master with the following error:
>>
>> $ sh bin/start-master.sh
>> starting org.apache.spark.deploy.master.Master, logging to
>> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>> failed to launch org.apache.spark.deploy.master.Master:
>>
>> Error: Could not find or load main class
>> org.apache.spark.deploy.master.Master
>> full log in
>> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>>
>> --Adrian (again. J)
>>
>>
>>
>
>
Re: Pre-build Spark for Windows 8.1
Posted by Matei Zaharia <ma...@gmail.com>.
Hey Andrew, unfortunately I don’t know how easy this is. Maybe future versions of Akka have it. We can certainly ask them to do it in general but I imagine there are some use cases where they wanted this behavior.
Matei
On Dec 5, 2013, at 2:49 PM, Andrew Ash <an...@andrewash.com> wrote:
> Speaking of akka and host sensitivity... How much have you hacked on akka to get it to support all of: myhost.mydomain.int, myhost, and 10.1.1.1? It's kind of a pain to get the Spark URL to exactly match. I'm wondering if there are usability gains that could be made here or if we're pretty stuck.
>
>
> On Thu, Dec 5, 2013 at 2:43 PM, Matei Zaharia <ma...@gmail.com> wrote:
> Hi,
>
> When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the URL (uppercase instead of lowercase). Unfortunately Akka is very specific about seeing hostnames written in the same way on each node, or else it thinks the message is for another machine!
>
> Matei
>
> On Dec 5, 2013, at 8:27 AM, Adrian Bonar <Ad...@microsoft.com> wrote:
>
>> The master starts up now as expected but the workers are unable to connect to the master. It looks like the master is refusing the connection messages but I’m not sure why. The first two error lines below are from trying to connect a worker from a separate machine and the last two error lines are from trying to connect a worker on the same machine as the master. I verified that the workers do not show up in the master’s web ui.
>>
>> MASTER:
>> D:\spark>spark-class org.apache.spark.deploy.master.Master
>> 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:08:34 INFO master.Master: Starting Spark master at spark://ADRIBONA-DEV-1:7077
>> 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/master/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:08:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8088
>> 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at http://ADRIBONA-DEV-1:8088
>> 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master atakka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
>>
>> WORKER:
>> D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker spark://adribona-dev-1:7077
>> 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
>> 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
>> 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/log,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/logPage,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:18:46 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8081
>> 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at http://ADRIBONA-DEV-1:8081
>> 13/12/05 08:18:46 INFO worker.Worker: Connecting to master spark://adribona-dev-1:7077
>>
>>
>> --Adrian
>>
>> From: Adrian Bonar [mailto:Adrian.Bonar@microsoft.com]
>> Sent: Thursday, December 5, 2013 7:49 AM
>> To: user@spark.incubator.apache.org
>> Subject: RE: Pre-build Spark for Windows 8.1
>>
>> Excellent! Thank you, Matei.
>>
>> From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
>> Sent: Wednesday, December 4, 2013 4:26 PM
>> To: user@spark.incubator.apache.org
>> Subject: Re: Pre-build Spark for Windows 8.1
>>
>> Hey Adrian,
>>
>> Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts we provide instead. Cygwin might be made to work but we haven’t tried to do this so far so it’s not supported. If you can fix it, that would of course be welcome.
>>
>> Also, the deploy scripts don’t work on Windows — we assumed that people would mostly use Windows for local development and testing. However, if you do want to launch a cluster, you can use spark-class.cmd manually to launch a Master and then Workers. Use the commands in the first part of http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
>>
>> Matei
>>
>> On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com> wrote:
>>
>>
>> Separate from my previous thread about building a distribution of Spark on Win8, I am also trying to run the pre-build binaries with little success. I downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to d:\spark and attempted to start a master with the following error:
>>
>> $ sh bin/start-master.sh
>> starting org.apache.spark.deploy.master.Master, logging to /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>> failed to launch org.apache.spark.deploy.master.Master:
>>
>> Error: Could not find or load main class org.apache.spark.deploy.master.Master
>> full log in /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>>
>> --Adrian (again. J)
>
>
Re: Pre-build Spark for Windows 8.1
Posted by Andrew Ash <an...@andrewash.com>.
Speaking of akka and host sensitivity... How much have you hacked on akka
to get it to support all of: myhost.mydomain.int, myhost, and 10.1.1.1?
It's kind of a pain to get the Spark URL to exactly match. I'm wondering
if there are usability gains that could be made here or if we're pretty
stuck.
On Thu, Dec 5, 2013 at 2:43 PM, Matei Zaharia <ma...@gmail.com>wrote:
> Hi,
>
> When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the
> URL (uppercase instead of lowercase). Unfortunately Akka is very specific
> about seeing hostnames written in the same way on each node, or else it
> thinks the message is for another machine!
>
> Matei
>
> On Dec 5, 2013, at 8:27 AM, Adrian Bonar <Ad...@microsoft.com>
> wrote:
>
> The master starts up now as expected but the workers are unable to
> connect to the master. It looks like the master is refusing the connection
> messages but I’m not sure why. The first two error lines below are from
> trying to connect a worker from a separate machine and the last two error
> lines are from trying to connect a worker on the same machine as the
> master. I verified that the workers do not show up in the master’s web ui.
>
> MASTER:
> D:\spark>spark-class org.apache.spark.deploy.master.Master
> 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
> 13/12/05 08:08:34 INFO master.Master: Starting Spark master at
> spark://ADRIBONA-DEV-1:7077
> 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
> 13/12/05 08:08:34 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/metrics/master/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/static,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/app/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/app,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{*,null}
> 13/12/05 08:08:34 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:8088
> 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at
> http://ADRIBONA-DEV-1:8088 <http://adribona-dev-1:8088/>
> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message
> RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1)
> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master
> at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
> akka://sparkMaster@ADRIBONA-DEV-1:7077
> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message
> DaemonMsgWatch(Actor[
> akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
> akka://sparkMaster@ADRIBONA-DEV-1:7077
> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message
> RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1)
> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master
> atakka://sparkMaster@ADRIBONA-DEV-1:7077 local is
> akka://sparkMaster@ADRIBONA-DEV-1:7077
> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message
> DaemonMsgWatch(Actor[
> akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
> akka://sparkMaster@ADRIBONA-DEV-1:7077
>
> WORKER:
> D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker
> spark://adribona-dev-1:7077
> 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
> 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker
> ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
> 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
> 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
> 13/12/05 08:18:46 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/metrics/json,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/static,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/log,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/logPage,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{/json,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started
> o.e.j.s.h.ContextHandler{*,null}
> 13/12/05 08:18:46 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:8081
> 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at
> http://ADRIBONA-DEV-1:8081 <http://adribona-dev-1:8081/>
> 13/12/05 08:18:46 INFO worker.Worker: Connecting to master
> spark://adribona-dev-1:7077
>
>
> --Adrian
>
> *From:* Adrian Bonar [mailto:Adrian.Bonar@microsoft.com<Ad...@microsoft.com>
> ]
> *Sent:* Thursday, December 5, 2013 7:49 AM
> *To:* user@spark.incubator.apache.org
> *Subject:* RE: Pre-build Spark for Windows 8.1
>
> Excellent! Thank you, Matei.
>
> *From:* Matei Zaharia [mailto:matei.zaharia@gmail.com<ma...@gmail.com>
> ]
> *Sent:* Wednesday, December 4, 2013 4:26 PM
> *To:* user@spark.incubator.apache.org
> *Subject:* Re: Pre-build Spark for Windows 8.1
>
> Hey Adrian,
>
> Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts
> we provide instead. Cygwin might be made to work but we haven’t tried to do
> this so far so it’s not supported. If you can fix it, that would of course
> be welcome.
>
> Also, the deploy scripts don’t work on Windows — we assumed that people
> would mostly use Windows for local development and testing. However, if you
> do want to launch a cluster, you can use spark-class.cmd manually to launch
> a Master and then Workers. Use the commands in the first part of
> http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
>
> Matei
>
> On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com>
> wrote:
>
>
>
> Separate from my previous thread about *building* a distribution of Spark
> on Win8, I am also trying to run the pre-build binaries with little
> success. I downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to
> d:\spark and attempted to start a master with the following error:
>
> $ sh bin/start-master.sh
> starting org.apache.spark.deploy.master.Master, logging to
> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
> failed to launch org.apache.spark.deploy.master.Master:
>
> Error: Could not find or load main class
> org.apache.spark.deploy.master.Master
> full log in
> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>
> --Adrian (again. J)
>
>
>
Re: Pre-build Spark for Windows 8.1
Posted by Matei Zaharia <ma...@gmail.com>.
Hi,
When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the URL (uppercase instead of lowercase). Unfortunately Akka is very specific about seeing hostnames written in the same way on each node, or else it thinks the message is for another machine!
Matei
On Dec 5, 2013, at 8:27 AM, Adrian Bonar <Ad...@microsoft.com> wrote:
> The master starts up now as expected but the workers are unable to connect to the master. It looks like the master is refusing the connection messages but I’m not sure why. The first two error lines below are from trying to connect a worker from a separate machine and the last two error lines are from trying to connect a worker on the same machine as the master. I verified that the workers do not show up in the master’s web ui.
>
> MASTER:
> D:\spark>spark-class org.apache.spark.deploy.master.Master
> 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
> 13/12/05 08:08:34 INFO master.Master: Starting Spark master at spark://ADRIBONA-DEV-1:7077
> 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/master/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
> 13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
> 13/12/05 08:08:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8088
> 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at http://ADRIBONA-DEV-1:8088
> 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
> 13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
> 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master atakka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
> 13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
>
> WORKER:
> D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker spark://adribona-dev-1:7077
> 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
> 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
> 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
> 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/json,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/log,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/logPage,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
> 13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
> 13/12/05 08:18:46 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8081
> 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at http://ADRIBONA-DEV-1:8081
> 13/12/05 08:18:46 INFO worker.Worker: Connecting to master spark://adribona-dev-1:7077
>
>
> --Adrian
>
> From: Adrian Bonar [mailto:Adrian.Bonar@microsoft.com]
> Sent: Thursday, December 5, 2013 7:49 AM
> To: user@spark.incubator.apache.org
> Subject: RE: Pre-build Spark for Windows 8.1
>
> Excellent! Thank you, Matei.
>
> From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
> Sent: Wednesday, December 4, 2013 4:26 PM
> To: user@spark.incubator.apache.org
> Subject: Re: Pre-build Spark for Windows 8.1
>
> Hey Adrian,
>
> Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts we provide instead. Cygwin might be made to work but we haven’t tried to do this so far so it’s not supported. If you can fix it, that would of course be welcome.
>
> Also, the deploy scripts don’t work on Windows — we assumed that people would mostly use Windows for local development and testing. However, if you do want to launch a cluster, you can use spark-class.cmd manually to launch a Master and then Workers. Use the commands in the first part of http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
>
> Matei
>
> On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com> wrote:
>
>
> Separate from my previous thread about building a distribution of Spark on Win8, I am also trying to run the pre-build binaries with little success. I downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to d:\spark and attempted to start a master with the following error:
>
> $ sh bin/start-master.sh
> starting org.apache.spark.deploy.master.Master, logging to /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
> failed to launch org.apache.spark.deploy.master.Master:
>
> Error: Could not find or load main class org.apache.spark.deploy.master.Master
> full log in /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>
> --Adrian (again. J)
RE: Pre-build Spark for Windows 8.1
Posted by Adrian Bonar <Ad...@microsoft.com>.
The master starts up now as expected but the workers are unable to connect to the master. It looks like the master is refusing the connection messages but I'm not sure why. The first two error lines below are from trying to connect a worker from a separate machine and the last two error lines are from trying to connect a worker on the same machine as the master. I verified that the workers do not show up in the master's web ui.
MASTER:
D:\spark>spark-class org.apache.spark.deploy.master.Master
13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/05 08:08:34 INFO master.Master: Starting Spark master at spark://ADRIBONA-DEV-1:7077
13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/master/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/app,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
13/12/05 08:08:34 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
13/12/05 08:08:34 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8088
13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at http://ADRIBONA-DEV-1:8088
13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:09:15 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/user/Master at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
13/12/05 08:18:46 ERROR NettyRemoteTransport(null): dropping message DaemonMsgWatch(Actor[akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master]) for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at akka://sparkMaster@ADRIBONA-DEV-1:7077 local is akka://sparkMaster@ADRIBONA-DEV-1:7077
WORKER:
D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker spark://adribona-dev-1:7077
13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/metrics/json,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/static,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/log,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/logPage,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{/json,null}
13/12/05 08:18:46 INFO handler.ContextHandler: started o.e.j.s.h.ContextHandler{*,null}
13/12/05 08:18:46 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8081
13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at http://ADRIBONA-DEV-1:8081
13/12/05 08:18:46 INFO worker.Worker: Connecting to master spark://adribona-dev-1:7077
--Adrian
From: Adrian Bonar [mailto:Adrian.Bonar@microsoft.com]
Sent: Thursday, December 5, 2013 7:49 AM
To: user@spark.incubator.apache.org
Subject: RE: Pre-build Spark for Windows 8.1
Excellent! Thank you, Matei.
From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
Sent: Wednesday, December 4, 2013 4:26 PM
To: user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>
Subject: Re: Pre-build Spark for Windows 8.1
Hey Adrian,
Ideally you shouldn't use Cygwin to run on Windows - use the .cmd scripts we provide instead. Cygwin might be made to work but we haven't tried to do this so far so it's not supported. If you can fix it, that would of course be welcome.
Also, the deploy scripts don't work on Windows - we assumed that people would mostly use Windows for local development and testing. However, if you do want to launch a cluster, you can use spark-class.cmd manually to launch a Master and then Workers. Use the commands in the first part of http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
Matei
On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com>> wrote:
Separate from my previous thread about building a distribution of Spark on Win8, I am also trying to run the pre-build binaries with little success. I downloaded and extract "spark-0.8.0-incubating-bin-hadoop1" to d:\spark and attempted to start a master with the following error:
$ sh bin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
failed to launch org.apache.spark.deploy.master.Master:
Error: Could not find or load main class org.apache.spark.deploy.master.Master
full log in /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
--Adrian (again. :))
RE: Pre-build Spark for Windows 8.1
Posted by Adrian Bonar <Ad...@microsoft.com>.
Excellent! Thank you, Matei.
From: Matei Zaharia [mailto:matei.zaharia@gmail.com]
Sent: Wednesday, December 4, 2013 4:26 PM
To: user@spark.incubator.apache.org
Subject: Re: Pre-build Spark for Windows 8.1
Hey Adrian,
Ideally you shouldn't use Cygwin to run on Windows - use the .cmd scripts we provide instead. Cygwin might be made to work but we haven't tried to do this so far so it's not supported. If you can fix it, that would of course be welcome.
Also, the deploy scripts don't work on Windows - we assumed that people would mostly use Windows for local development and testing. However, if you do want to launch a cluster, you can use spark-class.cmd manually to launch a Master and then Workers. Use the commands in the first part of http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
Matei
On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com>> wrote:
Separate from my previous thread about building a distribution of Spark on Win8, I am also trying to run the pre-build binaries with little success. I downloaded and extract "spark-0.8.0-incubating-bin-hadoop1" to d:\spark and attempted to start a master with the following error:
$ sh bin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
failed to launch org.apache.spark.deploy.master.Master:
Error: Could not find or load main class org.apache.spark.deploy.master.Master
full log in /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
--Adrian (again. :))
Re: Pre-build Spark for Windows 8.1
Posted by Matei Zaharia <ma...@gmail.com>.
Hey Adrian,
Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts we provide instead. Cygwin might be made to work but we haven’t tried to do this so far so it’s not supported. If you can fix it, that would of course be welcome.
Also, the deploy scripts don’t work on Windows — we assumed that people would mostly use Windows for local development and testing. However, if you do want to launch a cluster, you can use spark-class.cmd manually to launch a Master and then Workers. Use the commands in the first part of http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
Matei
On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Ad...@microsoft.com> wrote:
> Separate from my previous thread about building a distribution of Spark on Win8, I am also trying to run the pre-build binaries with little success. I downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to d:\spark and attempted to start a master with the following error:
>
> $ sh bin/start-master.sh
> starting org.apache.spark.deploy.master.Master, logging to /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
> failed to launch org.apache.spark.deploy.master.Master:
>
> Error: Could not find or load main class org.apache.spark.deploy.master.Master
> full log in /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>
> --Adrian (again. J)