You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by "Johannes Schillinger (Intern)" <jo...@citrix.com> on 2014/10/15 14:42:36 UTC
Connecting spark from a different Machine to mesos cluster
Hi,
we are currently trying to get a mesos cluster running as a base for Spark.
The mesos cluster itself runs and connecting a spark shell from the machine the maser runs on works perfectly.
We can see the Framework being started and the slaves working.
If we try to connect the exact same shell from a different machine to the exact same cluster the screen stops at
... 4013 sched.cpp:243] No credentials provided. Attempting to register without authentication
The cluster spins up a framework every two seconds with a new ID and stops it immediately. This continues (we stopped it after a few dozen starts).
We can see the frameworks being started in the master- and slave-logs as well as the command of the master to terminate it.
Has anyone ever encountered a similar problem or has any advice on solving this problem?
Thanks!
Johannes
RE: Connecting spark from a different Machine to mesos cluster
Posted by "Johannes Schillinger (Intern)" <jo...@citrix.com>.
Thank you all for your fast and extremely helpful replies!
Brians question concerning a network segment brought me to the idea to check my hosts-definition.
It was of course false, so changing my /etc/hosts – file to use my actual IP instead of 127.0.1.1 instantly solved this problem.
I hope this helps others with the same issue!
Thanks again!
Joahnes
From: Tim St Clair [mailto:tstclair@redhat.com]
Sent: Mittwoch, 15. Oktober 2014 19:17
To: user@mesos.apache.org
Subject: Re: Connecting spark from a different Machine to mesos cluster
Just as a general note, especially when dealing with VM's, NTP is *required* otherwise we've found you can get out of snyc'd updates which can result in some pretty strange behavior.
________________________________
From: "Brian Devins" <Br...@dealer.com>>
To: user@mesos.apache.org<ma...@mesos.apache.org>
Sent: Wednesday, October 15, 2014 11:57:19 AM
Subject: Re: Connecting spark from a different Machine to mesos cluster
Also Johannes, is there a network segment between Spark and the Mesos master? This looks like behavior I have seen before when the Master cannot connect back to the framework. The master also needs to be able to reach the Spark machine by IP
From: Tim Chen <ti...@mesosphere.io>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Wednesday, October 15, 2014 at 12:52 PM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Connecting spark from a different Machine to mesos cluster
Hi Johannes,
When you started your 2nd shell, what log output from the slave do you see for that framework?
Master seems to think it's already terminated.
Tim
On Wed, Oct 15, 2014 at 6:31 AM, Johannes Schillinger (Intern) <jo...@citrix.com>> wrote:
Hi Tim,
We are running Spark 1.1.0 with Hadoop 2.4. Mesos is in Version 0.20.1 all in binary releases.
The Spark console is running in default mode, which is fine grained.
The Spark process is started from a physical Machine running Ubuntu, the Mesos nodes are running in VMs also in Ubuntu.
This is the output of the Spark Shell:
--------------------------------------------------------------------------------------------------------------------------------
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/10/15 15:18:24 INFO SecurityManager: Changing view acls to: USERNAME,
14/10/15 15:18:24 INFO SecurityManager: Changing modify acls to: USERNAME,
14/10/15 15:18:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, )
14/10/15 15:18:24 INFO HttpServer: Starting HTTP Server
14/10/15 15:18:24 INFO Utils: Successfully started service 'HTTP class server' on port 42469.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
14/10/15 15:18:26 WARN Utils: Your hostname, karwjohannes01 resolves to a loopback address: 127.0.1.1; using CLIENT_IP instead (on interface eth0)
14/10/15 15:18:26 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/10/15 15:18:27 INFO SecurityManager: Changing view acls to: USERNAME,
14/10/15 15:18:27 INFO SecurityManager: Changing modify acls to: USERNAME,
14/10/15 15:18:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, )
14/10/15 15:18:27 INFO Slf4jLogger: Slf4jLogger started
14/10/15 15:18:27 INFO Remoting: Starting remoting
14/10/15 15:18:27 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@CLIENT_IP:51879]
14/10/15 15:18:27 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@CLIENT_IP:51879]
14/10/15 15:18:27 INFO Utils: Successfully started service 'sparkDriver' on port 51879.
14/10/15 15:18:27 INFO SparkEnv: Registering MapOutputTracker
14/10/15 15:18:27 INFO SparkEnv: Registering BlockManagerMaster
14/10/15 15:18:27 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141015151827-1a2e
14/10/15 15:18:27 INFO Utils: Successfully started service 'Connection manager for block manager' on port 60963.
14/10/15 15:18:27 INFO ConnectionManager: Bound socket to port 60963 with id = ConnectionManagerId(CLIENT_IP,60963)
14/10/15 15:18:27 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
14/10/15 15:18:27 INFO BlockManagerMaster: Trying to register BlockManager
14/10/15 15:18:27 INFO BlockManagerMasterActor: Registering block manager CLIENT_IP:60963 with 265.4 MB RAM
14/10/15 15:18:27 INFO BlockManagerMaster: Registered BlockManager
14/10/15 15:18:27 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b032c76c-93e1-473e-802c-c55e12e85d41
14/10/15 15:18:27 INFO HttpServer: Starting HTTP Server
14/10/15 15:18:27 INFO Utils: Successfully started service 'HTTP file server' on port 47989.
14/10/15 15:18:27 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/10/15 15:18:27 INFO SparkUI: Started SparkUI at http://CLIENT_IP:4040
14/10/15 15:18:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I1015 15:18:28.524736 4748 sched.cpp:139] Version: 0.20.1
I1015 15:18:28.527180 4750 sched.cpp:235] New master detected at master@MESOS_MASTER_IP:5050
I1015 15:18:28.527300 4750 sched.cpp:243] No credentials provided. Attempting to register without authentication
--------------------------------------------------------------------------------------------------------------------------------
Mesos master WARNING log:
W1015 14:13:00.235213 1118 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3490 because the framework has terminated or is inactive
W1015 14:13:35.244055 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3525 because the framework has terminated or is inactive
W1015 14:13:50.252436 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3540 because the framework has terminated or is inactive
W1015 14:14:05.252708 1117 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3555 because the framework has terminated or is inactive
Mesos slave WARNING log :
W1015 13:58:19.103196 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3116
W1015 13:58:20.104650 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3117
W1015 13:58:21.119839 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3118
W1015 13:58:22.115965 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3119
W1015 13:58:23.104925 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3120
W1015 13:58:24.104652 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3121
W1015 13:58:59.853744 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3122
W1015 13:59:00.853086 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3123
W1015 13:59:01.853137 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3124
W1015 13:59:03.318259 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3029
I hope this information helps, please ask if you have any more questions and thank you for your help!
Johannes
From: Tim St Clair [mailto:tstclair@redhat.com<ma...@redhat.com>]
Sent: Mittwoch, 15. Oktober 2014 15:11
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Connecting spark from a different Machine to mesos cluster
Details?
1. What versions are you running?
2. Fine grained mode or Course Gained?
3. Are you running in VM's?
Logs always help too.
Cheers,
Tim
________________________________
From: "Johannes Schillinger (Intern)" <jo...@citrix.com>>
To: user@mesos.apache.org<ma...@mesos.apache.org>
Sent: Wednesday, October 15, 2014 7:42:36 AM
Subject: Connecting spark from a different Machine to mesos cluster
Hi,
we are currently trying to get a mesos cluster running as a base for Spark.
The mesos cluster itself runs and connecting a spark shell from the machine the maser runs on works perfectly.
We can see the Framework being started and the slaves working.
If we try to connect the exact same shell from a different machine to the exact same cluster the screen stops at
… 4013 sched.cpp:243] No credentials provided. Attempting to register without authentication
The cluster spins up a framework every two seconds with a new ID and stops it immediately. This continues (we stopped it after a few dozen starts).
We can see the frameworks being started in the master- and slave-logs as well as the command of the master to terminate it.
Has anyone ever encountered a similar problem or has any advice on solving this problem?
Thanks!
Johannes
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Brian Devins | Java Developer
Brian.Devins@dealer.com<ma...@dealer.com>
[Dealer.com]
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Re: Connecting spark from a different Machine to mesos cluster
Posted by Tim St Clair <ts...@redhat.com>.
Just as a general note, especially when dealing with VM's, NTP is *required* otherwise we've found you can get out of snyc'd updates which can result in some pretty strange behavior.
----- Original Message -----
> From: "Brian Devins" <Br...@dealer.com>
> To: user@mesos.apache.org
> Sent: Wednesday, October 15, 2014 11:57:19 AM
> Subject: Re: Connecting spark from a different Machine to mesos cluster
> Also Johannes, is there a network segment between Spark and the Mesos master?
> This looks like behavior I have seen before when the Master cannot connect
> back to the framework. The master also needs to be able to reach the Spark
> machine by IP
> From: Tim Chen < tim@mesosphere.io >
> Reply-To: " user@mesos.apache.org " < user@mesos.apache.org >
> Date: Wednesday, October 15, 2014 at 12:52 PM
> To: " user@mesos.apache.org " < user@mesos.apache.org >
> Subject: Re: Connecting spark from a different Machine to mesos cluster
> Hi Johannes,
> When you started your 2nd shell, what log output from the slave do you see
> for that framework?
> Master seems to think it's already terminated.
> Tim
> On Wed, Oct 15, 2014 at 6:31 AM, Johannes Schillinger (Intern) <
> johannes.schillinger@citrix.com > wrote:
> > Hi Tim,
>
> > We are running Spark 1.1.0 with Hadoop 2.4. Mesos is in Version 0.20.1 all
> > in
> > binary releases.
>
> > The Spark console is running in default mode, which is fine grained.
>
> > The Spark process is started from a physical Machine running Ubuntu, the
> > Mesos nodes are running in VMs also in Ubuntu.
>
> > This is the output of the Spark Shell:
>
> > --------------------------------------------------------------------------------------------------------------------------------
>
> > Spark assembly has been built with Hive, including Datanucleus jars on
> > classpath
>
> > Using Spark's default log4j profile:
> > org/apache/spark/log4j-defaults.properties
>
> > 14/10/15 15:18:24 INFO SecurityManager: Changing view acls to: USERNAME,
>
> > 14/10/15 15:18:24 INFO SecurityManager: Changing modify acls to: USERNAME,
>
> > 14/10/15 15:18:24 INFO SecurityManager: SecurityManager: authentication
> > disabled; ui acls disabled; users with view permissions: Set(USERNAME, );
> > users with modify permissions: Set(USERNAME, )
>
> > 14/10/15 15:18:24 INFO HttpServer: Starting HTTP Server
>
> > 14/10/15 15:18:24 INFO Utils: Successfully started service 'HTTP class
> > server' on port 42469.
>
> > Welcome to
>
> > ____ __
>
> > / __/__ ___ _____/ /__
>
> > _\ \/ _ \/ _ `/ __/ '_/
>
> > /___/ .__/\_,_/_/ /_/\_\ version 1.1.0
>
> > /_/
>
> > Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65)
>
> > Type in expressions to have them evaluated.
>
> > Type :help for more information.
>
> > 14/10/15 15:18:26 WARN Utils: Your hostname, karwjohannes01 resolves to a
> > loopback address: 127.0.1.1; using CLIENT_IP instead (on interface eth0)
>
> > 14/10/15 15:18:26 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> > another address
>
> > 14/10/15 15:18:27 INFO SecurityManager: Changing view acls to: USERNAME,
>
> > 14/10/15 15:18:27 INFO SecurityManager: Changing modify acls to: USERNAME,
>
> > 14/10/15 15:18:27 INFO SecurityManager: SecurityManager: authentication
> > disabled; ui acls disabled; users with view permissions: Set(USERNAME, );
> > users with modify permissions: Set(USERNAME, )
>
> > 14/10/15 15:18:27 INFO Slf4jLogger: Slf4jLogger started
>
> > 14/10/15 15:18:27 INFO Remoting: Starting remoting
>
> > 14/10/15 15:18:27 INFO Remoting: Remoting started; listening on addresses
> > :[akka.tcp://sparkDriver@CLIENT_IP:51879]
>
> > 14/10/15 15:18:27 INFO Remoting: Remoting now listens on addresses:
> > [akka.tcp://sparkDriver@CLIENT_IP:51879]
>
> > 14/10/15 15:18:27 INFO Utils: Successfully started service 'sparkDriver' on
> > port 51879.
>
> > 14/10/15 15:18:27 INFO SparkEnv: Registering MapOutputTracker
>
> > 14/10/15 15:18:27 INFO SparkEnv: Registering BlockManagerMaster
>
> > 14/10/15 15:18:27 INFO DiskBlockManager: Created local directory at
> > /tmp/spark-local-20141015151827-1a2e
>
> > 14/10/15 15:18:27 INFO Utils: Successfully started service 'Connection
> > manager for block manager' on port 60963.
>
> > 14/10/15 15:18:27 INFO ConnectionManager: Bound socket to port 60963 with
> > id
> > = ConnectionManagerId(CLIENT_IP,60963)
>
> > 14/10/15 15:18:27 INFO MemoryStore: MemoryStore started with capacity 265.4
> > MB
>
> > 14/10/15 15:18:27 INFO BlockManagerMaster: Trying to register BlockManager
>
> > 14/10/15 15:18:27 INFO BlockManagerMasterActor: Registering block manager
> > CLIENT_IP:60963 with 265.4 MB RAM
>
> > 14/10/15 15:18:27 INFO BlockManagerMaster: Registered BlockManager
>
> > 14/10/15 15:18:27 INFO HttpFileServer: HTTP File server directory is
> > /tmp/spark-b032c76c-93e1-473e-802c-c55e12e85d41
>
> > 14/10/15 15:18:27 INFO HttpServer: Starting HTTP Server
>
> > 14/10/15 15:18:27 INFO Utils: Successfully started service 'HTTP file
> > server'
> > on port 47989.
>
> > 14/10/15 15:18:27 INFO Utils: Successfully started service 'SparkUI' on
> > port
> > 4040.
>
> > 14/10/15 15:18:27 INFO SparkUI: Started SparkUI at http://CLIENT_IP:4040
>
> > 14/10/15 15:18:27 WARN NativeCodeLoader: Unable to load native-hadoop
> > library
> > for your platform... using builtin-java classes where applicable
>
> > I1015 15:18:28.524736 4748 sched.cpp:139] Version: 0.20.1
>
> > I1015 15:18:28.527180 4750 sched.cpp:235] New master detected at
> > master@MESOS_MASTER_IP:5050
>
> > I1015 15:18:28.527300 4750 sched.cpp:243] No credentials provided.
> > Attempting
> > to register without authentication
>
> > --------------------------------------------------------------------------------------------------------------------------------
>
> > Mesos master WARNING log:
>
> > W1015 14:13:00.235213 1118 master.cpp:3452] Master returning resources
> > offered to framework 20141007-102213-343139338-5050-1037-3490 because the
> > framework has terminated or is inactive
>
> > W1015 14:13:35.244055 1121 master.cpp:3452] Master returning resources
> > offered to framework 20141007-102213-343139338-5050-1037-3525 because the
> > framework has terminated or is inactive
>
> > W1015 14:13:50.252436 1121 master.cpp:3452] Master returning resources
> > offered to framework 20141007-102213-343139338-5050-1037-3540 because the
> > framework has terminated or is inactive
>
> > W1015 14:14:05.252708 1117 master.cpp:3452] Master returning resources
> > offered to framework 20141007-102213-343139338-5050-1037-3555 because the
> > framework has terminated or is inactive
>
> > Mesos slave WARNING log :
>
> > W1015 13:58:19.103196 1211 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3116
>
> > W1015 13:58:20.104650 1210 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3117
>
> > W1015 13:58:21.119839 1211 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3118
>
> > W1015 13:58:22.115965 1210 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3119
>
> > W1015 13:58:23.104925 1211 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3120
>
> > W1015 13:58:24.104652 1210 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3121
>
> > W1015 13:58:59.853744 1212 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3122
>
> > W1015 13:59:00.853086 1214 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3123
>
> > W1015 13:59:01.853137 1212 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3124
>
> > W1015 13:59:03.318259 1214 slave.cpp:1421] Cannot shut down unknown
> > framework
> > 20141007-102213-343139338-5050-1037-3029
>
> > I hope this information helps, please ask if you have any more questions
> > and
> > thank you for your help!
>
> > Johannes
>
> > From: Tim St Clair [mailto: tstclair@redhat.com ]
>
> > Sent: Mittwoch, 15. Oktober 2014 15:11
>
> > To: user@mesos.apache.org
>
> > Subject: Re: Connecting spark from a different Machine to mesos cluster
>
> > Details?
>
> > 1. What versions are you running?
>
> > 2. Fine grained mode or Course Gained?
>
> > 3. Are you running in VM's?
>
> > Logs always help too.
>
> > Cheers,
>
> > Tim
>
> > > From: "Johannes Schillinger (Intern)" < johannes.schillinger@citrix.com >
> >
>
> > > To: user@mesos.apache.org
> >
>
> > > Sent: Wednesday, October 15, 2014 7:42:36 AM
> >
>
> > > Subject: Connecting spark from a different Machine to mesos cluster
> >
>
> > > Hi,
> >
>
> > > we are currently trying to get a mesos cluster running as a base for
> > > Spark.
> >
>
> > > The mesos cluster itself runs and connecting a spark shell from the
> > > machine
> > > the maser runs on works perfectly.
> >
>
> > > We can see the Framework being started and the slaves working.
> >
>
> > > If we try to connect the exact same shell from a different machine to the
> > > exact same cluster the screen stops at
> >
>
> > > … 4013 sched.cpp:243] No credentials provided. Attempting to register
> > > without
> > > authentication
> >
>
> > > The cluster spins up a framework every two seconds with a new ID and
> > > stops
> > > it
> > > immediately. This continues (we stopped it after a few dozen starts).
> >
>
> > > We can see the frameworks being started in the master- and slave-logs as
> > > well
> > > as the command of the master to terminate it.
> >
>
> > > Has anyone ever encountered a similar problem or has any advice on
> > > solving
> > > this problem?
> >
>
> > > Thanks!
> >
>
> > > Johannes
> >
>
> > --
>
> > Cheers,
>
> > Timothy St. Clair
>
> > Red Hat Inc.
>
> Brian Devins | Java Developer
> Brian.Devins@dealer.com
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Re: Connecting spark from a different Machine to mesos cluster
Posted by Vinod Kone <vi...@gmail.com>.
http://stackoverflow.com/questions/24559616/mesos-scheduler-slave-continuously-gets-disconnected
On Wed, Oct 15, 2014 at 9:57 AM, Brian Devins <Br...@dealer.com>
wrote:
> Also Johannes, is there a network segment between Spark and the Mesos
> master? This looks like behavior I have seen before when the Master cannot
> connect back to the framework. The master also needs to be able to reach
> the Spark machine by IP
>
> From: Tim Chen <ti...@mesosphere.io>
> Reply-To: "user@mesos.apache.org" <us...@mesos.apache.org>
> Date: Wednesday, October 15, 2014 at 12:52 PM
> To: "user@mesos.apache.org" <us...@mesos.apache.org>
>
> Subject: Re: Connecting spark from a different Machine to mesos cluster
>
> Hi Johannes,
>
> When you started your 2nd shell, what log output from the slave do you
> see for that framework?
>
> Master seems to think it's already terminated.
>
> Tim
>
> On Wed, Oct 15, 2014 at 6:31 AM, Johannes Schillinger (Intern) <
> johannes.schillinger@citrix.com> wrote:
>
>> Hi Tim,
>>
>>
>>
>> We are running Spark 1.1.0 with Hadoop 2.4. Mesos is in Version 0.20.1
>> all in binary releases.
>>
>>
>>
>> The Spark console is running in default mode, which is fine grained.
>>
>>
>>
>> The Spark process is started from a physical Machine running Ubuntu, the
>> Mesos nodes are running in VMs also in Ubuntu.
>>
>>
>>
>> This is the output of the Spark Shell:
>>
>>
>>
>>
>> --------------------------------------------------------------------------------------------------------------------------------
>>
>> Spark assembly has been built with Hive, including Datanucleus jars on
>> classpath
>>
>> Using Spark's default log4j profile:
>> org/apache/spark/log4j-defaults.properties
>>
>> 14/10/15 15:18:24 INFO SecurityManager: Changing view acls to: USERNAME,
>>
>> 14/10/15 15:18:24 INFO SecurityManager: Changing modify acls to: USERNAME,
>>
>> 14/10/15 15:18:24 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(USERNAME, );
>> users with modify permissions: Set(USERNAME, )
>>
>> 14/10/15 15:18:24 INFO HttpServer: Starting HTTP Server
>>
>> 14/10/15 15:18:24 INFO Utils: Successfully started service 'HTTP class
>> server' on port 42469.
>>
>> Welcome to
>>
>> ____ __
>>
>> / __/__ ___ _____/ /__
>>
>> _\ \/ _ \/ _ `/ __/ '_/
>>
>> /___/ .__/\_,_/_/ /_/\_\ version 1.1.0
>>
>> /_/
>>
>>
>>
>> Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65)
>>
>> Type in expressions to have them evaluated.
>>
>> Type :help for more information.
>>
>> 14/10/15 15:18:26 WARN Utils: Your hostname, karwjohannes01 resolves to a
>> loopback address: 127.0.1.1; using CLIENT_IP instead (on interface eth0)
>>
>> 14/10/15 15:18:26 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
>> another address
>>
>> 14/10/15 15:18:27 INFO SecurityManager: Changing view acls to: USERNAME,
>>
>> 14/10/15 15:18:27 INFO SecurityManager: Changing modify acls to: USERNAME,
>>
>> 14/10/15 15:18:27 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(USERNAME, );
>> users with modify permissions: Set(USERNAME, )
>>
>> 14/10/15 15:18:27 INFO Slf4jLogger: Slf4jLogger started
>>
>> 14/10/15 15:18:27 INFO Remoting: Starting remoting
>>
>> 14/10/15 15:18:27 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkDriver@CLIENT_IP:51879]
>>
>> 14/10/15 15:18:27 INFO Remoting: Remoting now listens on addresses:
>> [akka.tcp://sparkDriver@CLIENT_IP:51879]
>>
>> 14/10/15 15:18:27 INFO Utils: Successfully started service 'sparkDriver'
>> on port 51879.
>>
>> 14/10/15 15:18:27 INFO SparkEnv: Registering MapOutputTracker
>>
>> 14/10/15 15:18:27 INFO SparkEnv: Registering BlockManagerMaster
>>
>> 14/10/15 15:18:27 INFO DiskBlockManager: Created local directory at
>> /tmp/spark-local-20141015151827-1a2e
>>
>> 14/10/15 15:18:27 INFO Utils: Successfully started service 'Connection
>> manager for block manager' on port 60963.
>>
>> 14/10/15 15:18:27 INFO ConnectionManager: Bound socket to port 60963 with
>> id = ConnectionManagerId(CLIENT_IP,60963)
>>
>> 14/10/15 15:18:27 INFO MemoryStore: MemoryStore started with capacity
>> 265.4 MB
>>
>> 14/10/15 15:18:27 INFO BlockManagerMaster: Trying to register BlockManager
>>
>> 14/10/15 15:18:27 INFO BlockManagerMasterActor: Registering block manager
>> CLIENT_IP:60963 with 265.4 MB RAM
>>
>> 14/10/15 15:18:27 INFO BlockManagerMaster: Registered BlockManager
>>
>> 14/10/15 15:18:27 INFO HttpFileServer: HTTP File server directory is
>> /tmp/spark-b032c76c-93e1-473e-802c-c55e12e85d41
>>
>> 14/10/15 15:18:27 INFO HttpServer: Starting HTTP Server
>>
>> 14/10/15 15:18:27 INFO Utils: Successfully started service 'HTTP file
>> server' on port 47989.
>>
>> 14/10/15 15:18:27 INFO Utils: Successfully started service 'SparkUI' on
>> port 4040.
>>
>> 14/10/15 15:18:27 INFO SparkUI: Started SparkUI at http://CLIENT_IP:4040
>>
>> 14/10/15 15:18:27 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>>
>> I1015 15:18:28.524736 4748 sched.cpp:139] Version: 0.20.1
>>
>> I1015 15:18:28.527180 4750 sched.cpp:235] New master detected at
>> master@MESOS_MASTER_IP:5050
>>
>> I1015 15:18:28.527300 4750 sched.cpp:243] No credentials provided.
>> Attempting to register without authentication
>>
>>
>> --------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>> Mesos master WARNING log:
>>
>> W1015 14:13:00.235213 1118 master.cpp:3452] Master returning resources
>> offered to framework 20141007-102213-343139338-5050-1037-3490 because the
>> framework has terminated or is inactive
>>
>> W1015 14:13:35.244055 1121 master.cpp:3452] Master returning resources
>> offered to framework 20141007-102213-343139338-5050-1037-3525 because the
>> framework has terminated or is inactive
>>
>> W1015 14:13:50.252436 1121 master.cpp:3452] Master returning resources
>> offered to framework 20141007-102213-343139338-5050-1037-3540 because the
>> framework has terminated or is inactive
>>
>> W1015 14:14:05.252708 1117 master.cpp:3452] Master returning resources
>> offered to framework 20141007-102213-343139338-5050-1037-3555 because the
>> framework has terminated or is inactive
>>
>>
>>
>>
>>
>> Mesos slave WARNING log :
>>
>>
>>
>> W1015 13:58:19.103196 1211 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3116
>>
>> W1015 13:58:20.104650 1210 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3117
>>
>> W1015 13:58:21.119839 1211 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3118
>>
>> W1015 13:58:22.115965 1210 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3119
>>
>> W1015 13:58:23.104925 1211 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3120
>>
>> W1015 13:58:24.104652 1210 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3121
>>
>> W1015 13:58:59.853744 1212 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3122
>>
>> W1015 13:59:00.853086 1214 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3123
>>
>> W1015 13:59:01.853137 1212 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3124
>>
>> W1015 13:59:03.318259 1214 slave.cpp:1421] Cannot shut down unknown
>> framework 20141007-102213-343139338-5050-1037-3029
>>
>>
>>
>>
>>
>> I hope this information helps, please ask if you have any more questions
>> and thank you for your help!
>>
>>
>>
>> Johannes
>>
>>
>>
>> *From:* Tim St Clair [mailto:tstclair@redhat.com]
>> *Sent:* Mittwoch, 15. Oktober 2014 15:11
>> *To:* user@mesos.apache.org
>> *Subject:* Re: Connecting spark from a different Machine to mesos cluster
>>
>>
>>
>> Details?
>>
>>
>>
>> 1. What versions are you running?
>>
>> 2. Fine grained mode or Course Gained?
>>
>> 3. Are you running in VM's?
>>
>>
>>
>> Logs always help too.
>>
>>
>>
>> Cheers,
>>
>> Tim
>>
>>
>> ------------------------------
>>
>> *From: *"Johannes Schillinger (Intern)" <jo...@citrix.com>
>> *To: *user@mesos.apache.org
>> *Sent: *Wednesday, October 15, 2014 7:42:36 AM
>> *Subject: *Connecting spark from a different Machine to mesos cluster
>>
>>
>>
>> Hi,
>>
>>
>>
>> we are currently trying to get a mesos cluster running as a base for
>> Spark.
>>
>>
>>
>> The mesos cluster itself runs and connecting a spark shell from the
>> machine the maser runs on works perfectly.
>>
>> We can see the Framework being started and the slaves working.
>>
>>
>>
>> If we try to connect the exact same shell from a different machine to the
>> exact same cluster the screen stops at
>>
>>
>>
>> … 4013 sched.cpp:243] No credentials provided. Attempting to register
>> without authentication
>>
>>
>>
>> The cluster spins up a framework every two seconds with a new ID and
>> stops it immediately. This continues (we stopped it after a few dozen
>> starts).
>>
>>
>>
>> We can see the frameworks being started in the master- and slave-logs as
>> well as the command of the master to terminate it.
>>
>>
>>
>> Has anyone ever encountered a similar problem or has any advice on
>> solving this problem?
>>
>>
>>
>> Thanks!
>>
>> Johannes
>>
>>
>>
>>
>>
>> --
>>
>> Cheers,
>> Timothy St. Clair
>> Red Hat Inc.
>>
>
>
>
> Brian Devins* |* Java Developer
> Brian.Devins@dealer.com
>
> [image: Dealer.com]
>
>
Re: Connecting spark from a different Machine to mesos cluster
Posted by Brian Devins <Br...@dealer.com>.
Also Johannes, is there a network segment between Spark and the Mesos master? This looks like behavior I have seen before when the Master cannot connect back to the framework. The master also needs to be able to reach the Spark machine by IP
From: Tim Chen <ti...@mesosphere.io>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Wednesday, October 15, 2014 at 12:52 PM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: Connecting spark from a different Machine to mesos cluster
Hi Johannes,
When you started your 2nd shell, what log output from the slave do you see for that framework?
Master seems to think it's already terminated.
Tim
On Wed, Oct 15, 2014 at 6:31 AM, Johannes Schillinger (Intern) <jo...@citrix.com>> wrote:
Hi Tim,
We are running Spark 1.1.0 with Hadoop 2.4. Mesos is in Version 0.20.1 all in binary releases.
The Spark console is running in default mode, which is fine grained.
The Spark process is started from a physical Machine running Ubuntu, the Mesos nodes are running in VMs also in Ubuntu.
This is the output of the Spark Shell:
--------------------------------------------------------------------------------------------------------------------------------
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/10/15 15:18:24 INFO SecurityManager: Changing view acls to: USERNAME,
14/10/15 15:18:24 INFO SecurityManager: Changing modify acls to: USERNAME,
14/10/15 15:18:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, )
14/10/15 15:18:24 INFO HttpServer: Starting HTTP Server
14/10/15 15:18:24 INFO Utils: Successfully started service 'HTTP class server' on port 42469.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
14/10/15 15:18:26 WARN Utils: Your hostname, karwjohannes01 resolves to a loopback address: 127.0.1.1; using CLIENT_IP instead (on interface eth0)
14/10/15 15:18:26 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/10/15 15:18:27 INFO SecurityManager: Changing view acls to: USERNAME,
14/10/15 15:18:27 INFO SecurityManager: Changing modify acls to: USERNAME,
14/10/15 15:18:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, )
14/10/15 15:18:27 INFO Slf4jLogger: Slf4jLogger started
14/10/15 15:18:27 INFO Remoting: Starting remoting
14/10/15 15:18:27 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@CLIENT_IP:51879]
14/10/15 15:18:27 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@CLIENT_IP:51879]
14/10/15 15:18:27 INFO Utils: Successfully started service 'sparkDriver' on port 51879.
14/10/15 15:18:27 INFO SparkEnv: Registering MapOutputTracker
14/10/15 15:18:27 INFO SparkEnv: Registering BlockManagerMaster
14/10/15 15:18:27 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141015151827-1a2e
14/10/15 15:18:27 INFO Utils: Successfully started service 'Connection manager for block manager' on port 60963.
14/10/15 15:18:27 INFO ConnectionManager: Bound socket to port 60963 with id = ConnectionManagerId(CLIENT_IP,60963)
14/10/15 15:18:27 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
14/10/15 15:18:27 INFO BlockManagerMaster: Trying to register BlockManager
14/10/15 15:18:27 INFO BlockManagerMasterActor: Registering block manager CLIENT_IP:60963 with 265.4 MB RAM
14/10/15 15:18:27 INFO BlockManagerMaster: Registered BlockManager
14/10/15 15:18:27 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b032c76c-93e1-473e-802c-c55e12e85d41
14/10/15 15:18:27 INFO HttpServer: Starting HTTP Server
14/10/15 15:18:27 INFO Utils: Successfully started service 'HTTP file server' on port 47989.
14/10/15 15:18:27 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/10/15 15:18:27 INFO SparkUI: Started SparkUI at http://CLIENT_IP:4040
14/10/15 15:18:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I1015 15:18:28.524736 4748 sched.cpp:139] Version: 0.20.1
I1015 15:18:28.527180 4750 sched.cpp:235] New master detected at master@MESOS_MASTER_IP:5050
I1015 15:18:28.527300 4750 sched.cpp:243] No credentials provided. Attempting to register without authentication
--------------------------------------------------------------------------------------------------------------------------------
Mesos master WARNING log:
W1015 14:13:00.235213 1118 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3490 because the framework has terminated or is inactive
W1015 14:13:35.244055 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3525 because the framework has terminated or is inactive
W1015 14:13:50.252436 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3540 because the framework has terminated or is inactive
W1015 14:14:05.252708 1117 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3555 because the framework has terminated or is inactive
Mesos slave WARNING log :
W1015 13:58:19.103196 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3116
W1015 13:58:20.104650 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3117
W1015 13:58:21.119839 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3118
W1015 13:58:22.115965 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3119
W1015 13:58:23.104925 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3120
W1015 13:58:24.104652 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3121
W1015 13:58:59.853744 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3122
W1015 13:59:00.853086 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3123
W1015 13:59:01.853137 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3124
W1015 13:59:03.318259 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3029
I hope this information helps, please ask if you have any more questions and thank you for your help!
Johannes
From: Tim St Clair [mailto:tstclair@redhat.com<ma...@redhat.com>]
Sent: Mittwoch, 15. Oktober 2014 15:11
To: user@mesos.apache.org<ma...@mesos.apache.org>
Subject: Re: Connecting spark from a different Machine to mesos cluster
Details?
1. What versions are you running?
2. Fine grained mode or Course Gained?
3. Are you running in VM's?
Logs always help too.
Cheers,
Tim
________________________________
From: "Johannes Schillinger (Intern)" <jo...@citrix.com>>
To: user@mesos.apache.org<ma...@mesos.apache.org>
Sent: Wednesday, October 15, 2014 7:42:36 AM
Subject: Connecting spark from a different Machine to mesos cluster
Hi,
we are currently trying to get a mesos cluster running as a base for Spark.
The mesos cluster itself runs and connecting a spark shell from the machine the maser runs on works perfectly.
We can see the Framework being started and the slaves working.
If we try to connect the exact same shell from a different machine to the exact same cluster the screen stops at
… 4013 sched.cpp:243] No credentials provided. Attempting to register without authentication
The cluster spins up a framework every two seconds with a new ID and stops it immediately. This continues (we stopped it after a few dozen starts).
We can see the frameworks being started in the master- and slave-logs as well as the command of the master to terminate it.
Has anyone ever encountered a similar problem or has any advice on solving this problem?
Thanks!
Johannes
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Brian Devins | Java Developer
Brian.Devins@dealer.com<ma...@dealer.com>
[Dealer.com]
Re: Connecting spark from a different Machine to mesos cluster
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Johannes,
When you started your 2nd shell, what log output from the slave do you see
for that framework?
Master seems to think it's already terminated.
Tim
On Wed, Oct 15, 2014 at 6:31 AM, Johannes Schillinger (Intern) <
johannes.schillinger@citrix.com> wrote:
> Hi Tim,
>
>
>
> We are running Spark 1.1.0 with Hadoop 2.4. Mesos is in Version 0.20.1 all
> in binary releases.
>
>
>
> The Spark console is running in default mode, which is fine grained.
>
>
>
> The Spark process is started from a physical Machine running Ubuntu, the
> Mesos nodes are running in VMs also in Ubuntu.
>
>
>
> This is the output of the Spark Shell:
>
>
>
>
> --------------------------------------------------------------------------------------------------------------------------------
>
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> 14/10/15 15:18:24 INFO SecurityManager: Changing view acls to: USERNAME,
>
> 14/10/15 15:18:24 INFO SecurityManager: Changing modify acls to: USERNAME,
>
> 14/10/15 15:18:24 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(USERNAME, );
> users with modify permissions: Set(USERNAME, )
>
> 14/10/15 15:18:24 INFO HttpServer: Starting HTTP Server
>
> 14/10/15 15:18:24 INFO Utils: Successfully started service 'HTTP class
> server' on port 42469.
>
> Welcome to
>
> ____ __
>
> / __/__ ___ _____/ /__
>
> _\ \/ _ \/ _ `/ __/ '_/
>
> /___/ .__/\_,_/_/ /_/\_\ version 1.1.0
>
> /_/
>
>
>
> Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65)
>
> Type in expressions to have them evaluated.
>
> Type :help for more information.
>
> 14/10/15 15:18:26 WARN Utils: Your hostname, karwjohannes01 resolves to a
> loopback address: 127.0.1.1; using CLIENT_IP instead (on interface eth0)
>
> 14/10/15 15:18:26 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
>
> 14/10/15 15:18:27 INFO SecurityManager: Changing view acls to: USERNAME,
>
> 14/10/15 15:18:27 INFO SecurityManager: Changing modify acls to: USERNAME,
>
> 14/10/15 15:18:27 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(USERNAME, );
> users with modify permissions: Set(USERNAME, )
>
> 14/10/15 15:18:27 INFO Slf4jLogger: Slf4jLogger started
>
> 14/10/15 15:18:27 INFO Remoting: Starting remoting
>
> 14/10/15 15:18:27 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@CLIENT_IP:51879]
>
> 14/10/15 15:18:27 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkDriver@CLIENT_IP:51879]
>
> 14/10/15 15:18:27 INFO Utils: Successfully started service 'sparkDriver'
> on port 51879.
>
> 14/10/15 15:18:27 INFO SparkEnv: Registering MapOutputTracker
>
> 14/10/15 15:18:27 INFO SparkEnv: Registering BlockManagerMaster
>
> 14/10/15 15:18:27 INFO DiskBlockManager: Created local directory at
> /tmp/spark-local-20141015151827-1a2e
>
> 14/10/15 15:18:27 INFO Utils: Successfully started service 'Connection
> manager for block manager' on port 60963.
>
> 14/10/15 15:18:27 INFO ConnectionManager: Bound socket to port 60963 with
> id = ConnectionManagerId(CLIENT_IP,60963)
>
> 14/10/15 15:18:27 INFO MemoryStore: MemoryStore started with capacity
> 265.4 MB
>
> 14/10/15 15:18:27 INFO BlockManagerMaster: Trying to register BlockManager
>
> 14/10/15 15:18:27 INFO BlockManagerMasterActor: Registering block manager
> CLIENT_IP:60963 with 265.4 MB RAM
>
> 14/10/15 15:18:27 INFO BlockManagerMaster: Registered BlockManager
>
> 14/10/15 15:18:27 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-b032c76c-93e1-473e-802c-c55e12e85d41
>
> 14/10/15 15:18:27 INFO HttpServer: Starting HTTP Server
>
> 14/10/15 15:18:27 INFO Utils: Successfully started service 'HTTP file
> server' on port 47989.
>
> 14/10/15 15:18:27 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 14/10/15 15:18:27 INFO SparkUI: Started SparkUI at http://CLIENT_IP:4040
>
> 14/10/15 15:18:27 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> I1015 15:18:28.524736 4748 sched.cpp:139] Version: 0.20.1
>
> I1015 15:18:28.527180 4750 sched.cpp:235] New master detected at
> master@MESOS_MASTER_IP:5050
>
> I1015 15:18:28.527300 4750 sched.cpp:243] No credentials provided.
> Attempting to register without authentication
>
>
> --------------------------------------------------------------------------------------------------------------------------------
>
>
>
> Mesos master WARNING log:
>
> W1015 14:13:00.235213 1118 master.cpp:3452] Master returning resources
> offered to framework 20141007-102213-343139338-5050-1037-3490 because the
> framework has terminated or is inactive
>
> W1015 14:13:35.244055 1121 master.cpp:3452] Master returning resources
> offered to framework 20141007-102213-343139338-5050-1037-3525 because the
> framework has terminated or is inactive
>
> W1015 14:13:50.252436 1121 master.cpp:3452] Master returning resources
> offered to framework 20141007-102213-343139338-5050-1037-3540 because the
> framework has terminated or is inactive
>
> W1015 14:14:05.252708 1117 master.cpp:3452] Master returning resources
> offered to framework 20141007-102213-343139338-5050-1037-3555 because the
> framework has terminated or is inactive
>
>
>
>
>
> Mesos slave WARNING log :
>
>
>
> W1015 13:58:19.103196 1211 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3116
>
> W1015 13:58:20.104650 1210 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3117
>
> W1015 13:58:21.119839 1211 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3118
>
> W1015 13:58:22.115965 1210 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3119
>
> W1015 13:58:23.104925 1211 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3120
>
> W1015 13:58:24.104652 1210 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3121
>
> W1015 13:58:59.853744 1212 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3122
>
> W1015 13:59:00.853086 1214 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3123
>
> W1015 13:59:01.853137 1212 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3124
>
> W1015 13:59:03.318259 1214 slave.cpp:1421] Cannot shut down unknown
> framework 20141007-102213-343139338-5050-1037-3029
>
>
>
>
>
> I hope this information helps, please ask if you have any more questions
> and thank you for your help!
>
>
>
> Johannes
>
>
>
> *From:* Tim St Clair [mailto:tstclair@redhat.com]
> *Sent:* Mittwoch, 15. Oktober 2014 15:11
> *To:* user@mesos.apache.org
> *Subject:* Re: Connecting spark from a different Machine to mesos cluster
>
>
>
> Details?
>
>
>
> 1. What versions are you running?
>
> 2. Fine grained mode or Course Gained?
>
> 3. Are you running in VM's?
>
>
>
> Logs always help too.
>
>
>
> Cheers,
>
> Tim
>
>
> ------------------------------
>
> *From: *"Johannes Schillinger (Intern)" <jo...@citrix.com>
> *To: *user@mesos.apache.org
> *Sent: *Wednesday, October 15, 2014 7:42:36 AM
> *Subject: *Connecting spark from a different Machine to mesos cluster
>
>
>
> Hi,
>
>
>
> we are currently trying to get a mesos cluster running as a base for Spark.
>
>
>
> The mesos cluster itself runs and connecting a spark shell from the
> machine the maser runs on works perfectly.
>
> We can see the Framework being started and the slaves working.
>
>
>
> If we try to connect the exact same shell from a different machine to the
> exact same cluster the screen stops at
>
>
>
> … 4013 sched.cpp:243] No credentials provided. Attempting to register
> without authentication
>
>
>
> The cluster spins up a framework every two seconds with a new ID and stops
> it immediately. This continues (we stopped it after a few dozen starts).
>
>
>
> We can see the frameworks being started in the master- and slave-logs as
> well as the command of the master to terminate it.
>
>
>
> Has anyone ever encountered a similar problem or has any advice on solving
> this problem?
>
>
>
> Thanks!
>
> Johannes
>
>
>
>
>
> --
>
> Cheers,
> Timothy St. Clair
> Red Hat Inc.
>
RE: Connecting spark from a different Machine to mesos cluster
Posted by "Johannes Schillinger (Intern)" <jo...@citrix.com>.
Hi Tim,
We are running Spark 1.1.0 with Hadoop 2.4. Mesos is in Version 0.20.1 all in binary releases.
The Spark console is running in default mode, which is fine grained.
The Spark process is started from a physical Machine running Ubuntu, the Mesos nodes are running in VMs also in Ubuntu.
This is the output of the Spark Shell:
--------------------------------------------------------------------------------------------------------------------------------
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/10/15 15:18:24 INFO SecurityManager: Changing view acls to: USERNAME,
14/10/15 15:18:24 INFO SecurityManager: Changing modify acls to: USERNAME,
14/10/15 15:18:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, )
14/10/15 15:18:24 INFO HttpServer: Starting HTTP Server
14/10/15 15:18:24 INFO Utils: Successfully started service 'HTTP class server' on port 42469.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.1.0
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
14/10/15 15:18:26 WARN Utils: Your hostname, karwjohannes01 resolves to a loopback address: 127.0.1.1; using CLIENT_IP instead (on interface eth0)
14/10/15 15:18:26 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/10/15 15:18:27 INFO SecurityManager: Changing view acls to: USERNAME,
14/10/15 15:18:27 INFO SecurityManager: Changing modify acls to: USERNAME,
14/10/15 15:18:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(USERNAME, ); users with modify permissions: Set(USERNAME, )
14/10/15 15:18:27 INFO Slf4jLogger: Slf4jLogger started
14/10/15 15:18:27 INFO Remoting: Starting remoting
14/10/15 15:18:27 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@CLIENT_IP:51879]
14/10/15 15:18:27 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@CLIENT_IP:51879]
14/10/15 15:18:27 INFO Utils: Successfully started service 'sparkDriver' on port 51879.
14/10/15 15:18:27 INFO SparkEnv: Registering MapOutputTracker
14/10/15 15:18:27 INFO SparkEnv: Registering BlockManagerMaster
14/10/15 15:18:27 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20141015151827-1a2e
14/10/15 15:18:27 INFO Utils: Successfully started service 'Connection manager for block manager' on port 60963.
14/10/15 15:18:27 INFO ConnectionManager: Bound socket to port 60963 with id = ConnectionManagerId(CLIENT_IP,60963)
14/10/15 15:18:27 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
14/10/15 15:18:27 INFO BlockManagerMaster: Trying to register BlockManager
14/10/15 15:18:27 INFO BlockManagerMasterActor: Registering block manager CLIENT_IP:60963 with 265.4 MB RAM
14/10/15 15:18:27 INFO BlockManagerMaster: Registered BlockManager
14/10/15 15:18:27 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b032c76c-93e1-473e-802c-c55e12e85d41
14/10/15 15:18:27 INFO HttpServer: Starting HTTP Server
14/10/15 15:18:27 INFO Utils: Successfully started service 'HTTP file server' on port 47989.
14/10/15 15:18:27 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/10/15 15:18:27 INFO SparkUI: Started SparkUI at http://CLIENT_IP:4040
14/10/15 15:18:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
I1015 15:18:28.524736 4748 sched.cpp:139] Version: 0.20.1
I1015 15:18:28.527180 4750 sched.cpp:235] New master detected at master@MESOS_MASTER_IP:5050
I1015 15:18:28.527300 4750 sched.cpp:243] No credentials provided. Attempting to register without authentication
--------------------------------------------------------------------------------------------------------------------------------
Mesos master WARNING log:
W1015 14:13:00.235213 1118 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3490 because the framework has terminated or is inactive
W1015 14:13:35.244055 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3525 because the framework has terminated or is inactive
W1015 14:13:50.252436 1121 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3540 because the framework has terminated or is inactive
W1015 14:14:05.252708 1117 master.cpp:3452] Master returning resources offered to framework 20141007-102213-343139338-5050-1037-3555 because the framework has terminated or is inactive
Mesos slave WARNING log :
W1015 13:58:19.103196 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3116
W1015 13:58:20.104650 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3117
W1015 13:58:21.119839 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3118
W1015 13:58:22.115965 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3119
W1015 13:58:23.104925 1211 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3120
W1015 13:58:24.104652 1210 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3121
W1015 13:58:59.853744 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3122
W1015 13:59:00.853086 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3123
W1015 13:59:01.853137 1212 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3124
W1015 13:59:03.318259 1214 slave.cpp:1421] Cannot shut down unknown framework 20141007-102213-343139338-5050-1037-3029
I hope this information helps, please ask if you have any more questions and thank you for your help!
Johannes
From: Tim St Clair [mailto:tstclair@redhat.com]
Sent: Mittwoch, 15. Oktober 2014 15:11
To: user@mesos.apache.org
Subject: Re: Connecting spark from a different Machine to mesos cluster
Details?
1. What versions are you running?
2. Fine grained mode or Course Gained?
3. Are you running in VM's?
Logs always help too.
Cheers,
Tim
________________________________
From: "Johannes Schillinger (Intern)" <jo...@citrix.com>>
To: user@mesos.apache.org<ma...@mesos.apache.org>
Sent: Wednesday, October 15, 2014 7:42:36 AM
Subject: Connecting spark from a different Machine to mesos cluster
Hi,
we are currently trying to get a mesos cluster running as a base for Spark.
The mesos cluster itself runs and connecting a spark shell from the machine the maser runs on works perfectly.
We can see the Framework being started and the slaves working.
If we try to connect the exact same shell from a different machine to the exact same cluster the screen stops at
… 4013 sched.cpp:243] No credentials provided. Attempting to register without authentication
The cluster spins up a framework every two seconds with a new ID and stops it immediately. This continues (we stopped it after a few dozen starts).
We can see the frameworks being started in the master- and slave-logs as well as the command of the master to terminate it.
Has anyone ever encountered a similar problem or has any advice on solving this problem?
Thanks!
Johannes
--
Cheers,
Timothy St. Clair
Red Hat Inc.
Re: Connecting spark from a different Machine to mesos cluster
Posted by Tim St Clair <ts...@redhat.com>.
Details?
1. What versions are you running?
2. Fine grained mode or Course Gained?
3. Are you running in VM's?
Logs always help too.
Cheers,
Tim
----- Original Message -----
> From: "Johannes Schillinger (Intern)" <jo...@citrix.com>
> To: user@mesos.apache.org
> Sent: Wednesday, October 15, 2014 7:42:36 AM
> Subject: Connecting spark from a different Machine to mesos cluster
> Hi,
> we are currently trying to get a mesos cluster running as a base for Spark.
> The mesos cluster itself runs and connecting a spark shell from the machine
> the maser runs on works perfectly.
> We can see the Framework being started and the slaves working.
> If we try to connect the exact same shell from a different machine to the
> exact same cluster the screen stops at
> … 4013 sched.cpp:243] No credentials provided. Attempting to register without
> authentication
> The cluster spins up a framework every two seconds with a new ID and stops it
> immediately. This continues (we stopped it after a few dozen starts).
> We can see the frameworks being started in the master- and slave-logs as well
> as the command of the master to terminate it.
> Has anyone ever encountered a similar problem or has any advice on solving
> this problem?
> Thanks!
> Johannes
--
Cheers,
Timothy St. Clair
Red Hat Inc.