You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Stefano Bianchi <ja...@gmail.com> on 2016/04/14 09:33:14 UTC
Mesos-DNS Failed to connect to...
Im correctly running mesos-dns from marathon and it seems to work.
But when i launch:
http://test.marathon.mesos
(where test is a funning task on marathon)
I get:
curl: (7) Failed connect to test.marathon.mesos:80; Connection refused
Where am i wrong?
Il 13/apr/2016 17:46, "June Taylor" <ju...@umn.edu> ha scritto:
> We are running pyspark against our cluster in coarse-grained mode by
> specifying the --master mesos://host:5050 flag, which properly creates
> one task on each node.
>
> However, if the driver is shut down, it appears that these executors
> become orphaned_tasks, still consuming resources on the slave, but no
> longer being represented in the master's understanding of available
> resources.
>
> Examining the stdout/stderr shows it exited:
>
> Registered executor on node4
> Starting task 0
> sh -c 'cd spark-1*; ./bin/spark-class
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://
> CoarseGrainedScheduler@128.101.163.200:41563 --executor-id
> aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0 --hostname node4 --cores 31
> --app-id aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
> Forked command at 117620
> Command exited with status 1 (pid: 117620)
>
> But, these executors are remaining on all the slaves.
>
> What can we do to clear them out? Stopping mesos-slave and removing the
> full work-dir is successful, but also destroys our other tasks.
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
Re: Mesos-DNS Failed to connect to...
Posted by shakeel <sh...@motortrak.com>.
Hi Stephano,
In this case you should follow Chris advice of putting the mesos-dns
server as your local dns server.
It will allow you to connect to the services using a URL.
Kind Regards
Shakeel Suffee
On 14/04/16 14:29, Stefano Bianchi wrote:
> shakeel
>
> Dig command are perfectly working, i see the correct address on which i
> have mesos-dns running
> Unfortunately in the openstack environment where i am working there is
> not a DNS.
> And this miss it's causing me a lot a issues also for other stuff.
>
> 2016-04-14 15:22 GMT+02:00 shakeel <shakeel.suffee@motortrak.com
> <ma...@motortrak.com>>:
>
> Hi,
>
> Once you have mesos-dns running from marathon, test that it's working
> properly with dig.
>
> (You migth want to add you main dns servers as resolvers within the
> mesos-dns config and allow recursion.)
>
> Otherwise, configure you slaves to use the mesos-dns as their dns
> servers.
>
> I created a subdomain on my main dns server for the mesos-dns subdomain
> which points to the slave where mesos-dns is running.
>
> This way when you try to access the mesos-dns url from your browser, you
> main dns server will know where to forward the request.
>
> HTH.
>
> Kind Regards
> Shakeel Suffee
>
> On 14/04/16 14:06, Chris Baker wrote:
> > Also, make sure that the machine you're trying to launch from has
> > Mesos-DNS as its DNS server :)
> >
> > On Thu, Apr 14, 2016 at 3:33 AM Stefano Bianchi <jazzista88@gmail.com <ma...@gmail.com>
> > <mailto:jazzista88@gmail.com <ma...@gmail.com>>> wrote:
> >
> > Im correctly running mesos-dns from marathon and it seems to work.
> > But when i launch:
> >
> > http://test.marathon.mesos
> >
> > (where test is a funning task on marathon)
> >
> > I get:
> >
> > curl: (7) Failed connect to test.marathon.mesos:80; Connection refused
> >
> > Where am i wrong?
> >
> > Il 13/apr/2016 17:46, "June Taylor" <june@umn.edu <ma...@umn.edu>
> > <mailto:june@umn.edu <ma...@umn.edu>>> ha scritto:
> >
> > We are running pyspark against our cluster in coarse-grained
> > mode by specifying the --master mesos://host:5050 flag, which
> > properly creates one task on each node.
> >
> > However, if the driver is shut down, it appears that these
> > executors become orphaned_tasks, still consuming resources on
> > the slave, but no longer being represented in the master's
> > understanding of available resources.
> >
> > Examining the stdout/stderr shows it exited:
> >
> > Registered executor on node4
> > Starting task 0
> > sh -c 'cd spark-1*; ./bin/spark-class
> > org.apache.spark.executor.CoarseGrainedExecutorBackend
> > --driver-url
> > spark://CoarseGrainedScheduler@128.101.163.200:41563
> <http://CoarseGrainedScheduler@128.101.163.200:41563>
> > <http://CoarseGrainedScheduler@128.101.163.200:41563>
> > --executor-id aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0
> > --hostname node4 --cores 31 --app-id
> > aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
> > Forked command at 117620
> > Command exited with status 1 (pid: 117620)
> >
> > But, these executors are remaining on all the slaves.
> >
> > What can we do to clear them out? Stopping mesos-slave and
> > removing the full work-dir is successful, but also
> destroys our
> > other tasks.
> >
> > Thanks,
> > June Taylor
> > System Administrator, Minnesota Population Center
> > University of Minnesota
> >
>
> --
> The information contained in this message is for the intended addressee
> only and may contain confidential and/or privileged information. If
> you are
> not the intended addressee, please delete this message and notify the
> sender; do not copy or distribute this message or disclose its
> contents to
> anyone. Any views or opinions expressed in this message are those of the
> author and do not necessarily represent those of Motortrak Limited or of
> any of its associated companies. No reliance may be placed on this
> message
> without written confirmation from an authorised representative of the
> company.
>
> Registered in England 3098391 V.A.T. Registered No. 667463890
>
>
--
The information contained in this message is for the intended addressee
only and may contain confidential and/or privileged information. If you are
not the intended addressee, please delete this message and notify the
sender; do not copy or distribute this message or disclose its contents to
anyone. Any views or opinions expressed in this message are those of the
author and do not necessarily represent those of Motortrak Limited or of
any of its associated companies. No reliance may be placed on this message
without written confirmation from an authorised representative of the
company.
Registered in England 3098391 V.A.T. Registered No. 667463890
Re: Mesos-DNS Failed to connect to...
Posted by Stefano Bianchi <ja...@gmail.com>.
shakeel
Dig command are perfectly working, i see the correct address on which i
have mesos-dns running
Unfortunately in the openstack environment where i am working there is not
a DNS.
And this miss it's causing me a lot a issues also for other stuff.
2016-04-14 15:22 GMT+02:00 shakeel <sh...@motortrak.com>:
> Hi,
>
> Once you have mesos-dns running from marathon, test that it's working
> properly with dig.
>
> (You migth want to add you main dns servers as resolvers within the
> mesos-dns config and allow recursion.)
>
> Otherwise, configure you slaves to use the mesos-dns as their dns servers.
>
> I created a subdomain on my main dns server for the mesos-dns subdomain
> which points to the slave where mesos-dns is running.
>
> This way when you try to access the mesos-dns url from your browser, you
> main dns server will know where to forward the request.
>
> HTH.
>
> Kind Regards
> Shakeel Suffee
>
> On 14/04/16 14:06, Chris Baker wrote:
> > Also, make sure that the machine you're trying to launch from has
> > Mesos-DNS as its DNS server :)
> >
> > On Thu, Apr 14, 2016 at 3:33 AM Stefano Bianchi <jazzista88@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Im correctly running mesos-dns from marathon and it seems to work.
> > But when i launch:
> >
> > http://test.marathon.mesos
> >
> > (where test is a funning task on marathon)
> >
> > I get:
> >
> > curl: (7) Failed connect to test.marathon.mesos:80; Connection
> refused
> >
> > Where am i wrong?
> >
> > Il 13/apr/2016 17:46, "June Taylor" <june@umn.edu
> > <ma...@umn.edu>> ha scritto:
> >
> > We are running pyspark against our cluster in coarse-grained
> > mode by specifying the --master mesos://host:5050 flag, which
> > properly creates one task on each node.
> >
> > However, if the driver is shut down, it appears that these
> > executors become orphaned_tasks, still consuming resources on
> > the slave, but no longer being represented in the master's
> > understanding of available resources.
> >
> > Examining the stdout/stderr shows it exited:
> >
> > Registered executor on node4
> > Starting task 0
> > sh -c 'cd spark-1*; ./bin/spark-class
> > org.apache.spark.executor.CoarseGrainedExecutorBackend
> > --driver-url
> > spark://CoarseGrainedScheduler@128.101.163.200:41563
> > <http://CoarseGrainedScheduler@128.101.163.200:41563>
> > --executor-id aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0
> > --hostname node4 --cores 31 --app-id
> > aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
> > Forked command at 117620
> > Command exited with status 1 (pid: 117620)
> >
> > But, these executors are remaining on all the slaves.
> >
> > What can we do to clear them out? Stopping mesos-slave and
> > removing the full work-dir is successful, but also destroys our
> > other tasks.
> >
> > Thanks,
> > June Taylor
> > System Administrator, Minnesota Population Center
> > University of Minnesota
> >
>
> --
> The information contained in this message is for the intended addressee
> only and may contain confidential and/or privileged information. If you are
> not the intended addressee, please delete this message and notify the
> sender; do not copy or distribute this message or disclose its contents to
> anyone. Any views or opinions expressed in this message are those of the
> author and do not necessarily represent those of Motortrak Limited or of
> any of its associated companies. No reliance may be placed on this message
> without written confirmation from an authorised representative of the
> company.
>
> Registered in England 3098391 V.A.T. Registered No. 667463890
>
Re: Mesos-DNS Failed to connect to...
Posted by shakeel <sh...@motortrak.com>.
Hi,
Once you have mesos-dns running from marathon, test that it's working
properly with dig.
(You migth want to add you main dns servers as resolvers within the
mesos-dns config and allow recursion.)
Otherwise, configure you slaves to use the mesos-dns as their dns servers.
I created a subdomain on my main dns server for the mesos-dns subdomain
which points to the slave where mesos-dns is running.
This way when you try to access the mesos-dns url from your browser, you
main dns server will know where to forward the request.
HTH.
Kind Regards
Shakeel Suffee
On 14/04/16 14:06, Chris Baker wrote:
> Also, make sure that the machine you're trying to launch from has
> Mesos-DNS as its DNS server :)
>
> On Thu, Apr 14, 2016 at 3:33 AM Stefano Bianchi <jazzista88@gmail.com
> <ma...@gmail.com>> wrote:
>
> Im correctly running mesos-dns from marathon and it seems to work.
> But when i launch:
>
> http://test.marathon.mesos
>
> (where test is a funning task on marathon)
>
> I get:
>
> curl: (7) Failed connect to test.marathon.mesos:80; Connection refused
>
> Where am i wrong?
>
> Il 13/apr/2016 17:46, "June Taylor" <june@umn.edu
> <ma...@umn.edu>> ha scritto:
>
> We are running pyspark against our cluster in coarse-grained
> mode by specifying the --master mesos://host:5050 flag, which
> properly creates one task on each node.
>
> However, if the driver is shut down, it appears that these
> executors become orphaned_tasks, still consuming resources on
> the slave, but no longer being represented in the master's
> understanding of available resources.
>
> Examining the stdout/stderr shows it exited:
>
> Registered executor on node4
> Starting task 0
> sh -c 'cd spark-1*; ./bin/spark-class
> org.apache.spark.executor.CoarseGrainedExecutorBackend
> --driver-url
> spark://CoarseGrainedScheduler@128.101.163.200:41563
> <http://CoarseGrainedScheduler@128.101.163.200:41563>
> --executor-id aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0
> --hostname node4 --cores 31 --app-id
> aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
> Forked command at 117620
> Command exited with status 1 (pid: 117620)
>
> But, these executors are remaining on all the slaves.
>
> What can we do to clear them out? Stopping mesos-slave and
> removing the full work-dir is successful, but also destroys our
> other tasks.
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
--
The information contained in this message is for the intended addressee
only and may contain confidential and/or privileged information. If you are
not the intended addressee, please delete this message and notify the
sender; do not copy or distribute this message or disclose its contents to
anyone. Any views or opinions expressed in this message are those of the
author and do not necessarily represent those of Motortrak Limited or of
any of its associated companies. No reliance may be placed on this message
without written confirmation from an authorised representative of the
company.
Registered in England 3098391 V.A.T. Registered No. 667463890
Re: Mesos-DNS Failed to connect to...
Posted by Chris Baker <ch...@galacticfog.com>.
Also, make sure that the machine you're trying to launch from has Mesos-DNS
as its DNS server :)
On Thu, Apr 14, 2016 at 3:33 AM Stefano Bianchi <ja...@gmail.com>
wrote:
> Im correctly running mesos-dns from marathon and it seems to work.
> But when i launch:
>
> http://test.marathon.mesos
>
> (where test is a funning task on marathon)
>
> I get:
>
> curl: (7) Failed connect to test.marathon.mesos:80; Connection refused
>
> Where am i wrong?
> Il 13/apr/2016 17:46, "June Taylor" <ju...@umn.edu> ha scritto:
>
>> We are running pyspark against our cluster in coarse-grained mode by
>> specifying the --master mesos://host:5050 flag, which properly creates
>> one task on each node.
>>
>> However, if the driver is shut down, it appears that these executors
>> become orphaned_tasks, still consuming resources on the slave, but no
>> longer being represented in the master's understanding of available
>> resources.
>>
>> Examining the stdout/stderr shows it exited:
>>
>> Registered executor on node4
>> Starting task 0
>> sh -c 'cd spark-1*; ./bin/spark-class
>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://
>> CoarseGrainedScheduler@128.101.163.200:41563 --executor-id
>> aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0 --hostname node4 --cores 31
>> --app-id aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
>> Forked command at 117620
>> Command exited with status 1 (pid: 117620)
>>
>> But, these executors are remaining on all the slaves.
>>
>> What can we do to clear them out? Stopping mesos-slave and removing the
>> full work-dir is successful, but also destroys our other tasks.
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>
Re: Mesos-DNS Failed to connect to...
Posted by Rodrick Brown <ro...@orchard-app.com>.
https://mesosphere.github.io/mesos-dns/docs/naming.html
\--
**Rodrick Brown** / Systems Engineer
+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
**Orchard Platform**
101 5th Avenue, 4th Floor, New York, NY 10003
[http://www.orchardplatform.com](http://www.orchardplatform.com/)
[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
On Apr 14 2016, at 3:33 am, Stefano Bianchi <jazzista88@gmail.com>
wrote:
> Im correctly running mesos-dns from marathon and it seems to work.
But when i launch:
>
> <http://test.marathon.mesos>
>
> (where test is a funning task on marathon)
>
> I get:
>
> curl: (7) Failed connect to test.marathon.mesos:80; Connection refused
>
> Where am i wrong?
>
> Il 13/apr/2016 17:46, "June Taylor"
<[june@umn.edu](mailto:june@umn.edu)> ha scritto:
>
>> We are running pyspark against our cluster in coarse-grained mode by
specifying the \--master mesos://host:5050 flag, which properly creates one
task on each node.
>>
>>
>>
>> However, if the driver is shut down, it appears that these executors become
orphaned_tasks, still consuming resources on the slave, but no longer being
represented in the master's understanding of available resources.
>>
>>
>>
>> Examining the stdout/stderr shows it exited:
>>
>>
>>
>> Registered executor on node4
>>
>> Starting task 0
>>
>> sh -c 'cd spark-1*; ./bin/spark-class
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://[C
oarseGrainedScheduler@128.101.163.200:41563](http://CoarseGrainedScheduler@128
.101.163.200:41563) \--executor-id aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0
--hostname node4 --cores 31 --app-id
aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
>>
>> Forked command at 117620
>>
>> Command exited with status 1 (pid: 117620)
>>
>>
>>
>> But, these executors are remaining on all the slaves.
>>
>>
>>
>> What can we do to clear them out? Stopping mesos-slave and removing the
full work-dir is successful, but also destroys our other tasks.
>>
>>
>>
>> Thanks,
>>
>> June Taylor
>>
>> System Administrator, Minnesota Population Center
>>
>> University of Minnesota
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: Mesos-DNS Failed to connect to...
Posted by June Taylor <ju...@umn.edu>.
Stefano,
Try inspecting the DNS directly, for example here is an nslookup query to
find the port and slave node that contains a running Docker container
started by Marathon, and then you can see the curl command touching on that
node and the port specified in the SRV record. I am not sure your
expectation of having test.marathon.mesos as a valid DNS A-record is
correct.
june@cluster:~$ nslookup -type=SRV _tomsflask._tcp.marathon.mesos
_tomsflask._tcp.marathon.mesos service = 0 0 31427 tomsflask-5p8ho-s83.
marathon.slave.mesos.
june@cluster:~$ curl http://tomsflask-5p8ho-s83.marathon.slave.mesos:31427
Hello World from Flask (default)
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Thu, Apr 14, 2016 at 2:33 AM, Stefano Bianchi <ja...@gmail.com>
wrote:
> Im correctly running mesos-dns from marathon and it seems to work.
> But when i launch:
>
> http://test.marathon.mesos
>
> (where test is a funning task on marathon)
>
> I get:
>
> curl: (7) Failed connect to test.marathon.mesos:80; Connection refused
>
> Where am i wrong?
> Il 13/apr/2016 17:46, "June Taylor" <ju...@umn.edu> ha scritto:
>
>> We are running pyspark against our cluster in coarse-grained mode by
>> specifying the --master mesos://host:5050 flag, which properly creates
>> one task on each node.
>>
>> However, if the driver is shut down, it appears that these executors
>> become orphaned_tasks, still consuming resources on the slave, but no
>> longer being represented in the master's understanding of available
>> resources.
>>
>> Examining the stdout/stderr shows it exited:
>>
>> Registered executor on node4
>> Starting task 0
>> sh -c 'cd spark-1*; ./bin/spark-class
>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://
>> CoarseGrainedScheduler@128.101.163.200:41563 --executor-id
>> aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0 --hostname node4 --cores 31
>> --app-id aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
>> Forked command at 117620
>> Command exited with status 1 (pid: 117620)
>>
>> But, these executors are remaining on all the slaves.
>>
>> What can we do to clear them out? Stopping mesos-slave and removing the
>> full work-dir is successful, but also destroys our other tasks.
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>