You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sebastian Kuepers <se...@publicispixelpark.de> on 2015/11/02 15:24:20 UTC
Spark, Mesos problems with remote connections
Hey,
I have a Mesos cluster with a single Master. If I run the following directly on the master machine:
pyspark --master mesos://host:5050
everything works just fine. If I try to connect from to the master starting a driver from my laptop everything stops after the following log output from the Spark driver:
I1102 15:10:57.848831 64856064 sched.cpp:164] Version: 0.24.0
I1102 15:10:57.852708 19017728 sched.cpp:262] New master detected at master@217.66.55.19:5050
I1102 15:10:57.852934 19017728 sched.cpp:272] No credentials provided. Attempting to register without authentication
The Mesos master logs show the following:
I1102 15:09:50.676004 21563 master.cpp:2250] Subscribing framework Talos with checkpointing disabled and capabilities [ ]
I1102 15:09:50.676686 21566 hierarchical.hpp:515] Added framework b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004
I1102 15:09:51.671437 21562 http.cpp:336] HTTP GET for /master/state.json from 217.66.51.150:51588 with User-Agent='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36'
I1102 15:09:52.357230 21567 master.cpp:2179] Received SUBSCRIBE call for framework 'Talos' at scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
I1102 15:09:52.357416 21567 master.cpp:2250] Subscribing framework Talos with checkpointing disabled and capabilities [ ]
I1102 15:09:52.357520 21567 master.cpp:2260] Framework b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004 (Talos) at scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085 already subscribed, resending acknowledgement
I1102 15:09:53.938227 21563 master.cpp:2179] Received SUBSCRIBE call for framework 'Talos' at scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
I1102 15:09:53.938426 21563 master.cpp:2250] Subscribing framework Talos with checkpointing disabled and capabilities [ ]
I1102 15:09:53.938534 21563 master.cpp:2260] Framework b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004 (Talos) at scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085 already subscribed, resending acknowledgement
I1102 15:10:00.207372 21567 master.cpp:2179] Received SUBSCRIBE call for framework 'Talos' at scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
The framework also shows up as active in the UI and blocks all resources of the cluster, while the SUBSCRIBE calls keep coming in.
Mesos authentication is completely disabled.
What could be possible causes for this problem?
Thanks,
Sebastian
------------------------------------------------------------------------
Disclaimer The information in this email and any attachments may contain proprietary and confidential information that is intended for the addressee(s) only. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, retention or use of the contents of this information is prohibited. When addressed to our clients or vendors, any information contained in this e-mail or any attachments is subject to the terms and conditions in any governing contract. If you have received this e-mail in error, please immediately contact the sender and delete the e-mail.
Re: Spark, Mesos problems with remote connections
Posted by Tamas Szuromi <ta...@odigeo.com>.
Hello Sebastian,
Did you set the MESOS_NATIVE_JAVA_LIBRARY variable before you started
pyspark?
cheers,
Tamas
On 2 November 2015 at 15:24, Sebastian Kuepers <
sebastian.kuepers@publicispixelpark.de> wrote:
> Hey,
>
>
> I have a Mesos cluster with a single Master. If I run the following
> directly on the master machine:
>
>
> pyspark --master mesos://host:5050
>
>
> everything works just fine. If I try to connect from to the master
> starting a driver from my laptop everything stops after the following
> log output from the Spark driver:
>
>
> I1102 15:10:57.848831 64856064 sched.cpp:164] Version: 0.24.0
> I1102 15:10:57.852708 19017728 sched.cpp:262] New master detected at
> master@217.66.55.19:5050
> I1102 15:10:57.852934 19017728 sched.cpp:272] No credentials provided.
> Attempting to register without authentication
>
>
> The Mesos master logs show the following:
>
>
> I1102 15:09:50.676004 21563 master.cpp:2250] Subscribing framework Talos
> with checkpointing disabled and capabilities [ ]
> I1102 15:09:50.676686 21566 hierarchical.hpp:515] Added framework
> b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004
> I1102 15:09:51.671437 21562 http.cpp:336] HTTP GET for /master/state.json
> from 217.66.51.150:51588 with User-Agent='Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36'
> I1102 15:09:52.357230 21567 master.cpp:2179] Received SUBSCRIBE call for
> framework 'Talos' at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> I1102 15:09:52.357416 21567 master.cpp:2250] Subscribing framework Talos
> with checkpointing disabled and capabilities [ ]
> I1102 15:09:52.357520 21567 master.cpp:2260] Framework
> b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004 (Talos) at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> already subscribed, resending acknowledgement
> I1102 15:09:53.938227 21563 master.cpp:2179] Received SUBSCRIBE call for
> framework 'Talos' at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> I1102 15:09:53.938426 21563 master.cpp:2250] Subscribing framework Talos
> with checkpointing disabled and capabilities [ ]
> I1102 15:09:53.938534 21563 master.cpp:2260] Framework
> b995b024-6ec6-4bd9-b251-e4e6b1828eda-0004 (Talos) at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
> already subscribed, resending acknowledgement
> I1102 15:10:00.207372 21567 master.cpp:2179] Received SUBSCRIBE call for
> framework 'Talos' at
> scheduler-fee376b3-ea04-4866-8412-957721da5a0b@10.200.120.148:51085
>
>
> The framework also shows up as active in the UI and blocks all resources
> of the cluster, while the SUBSCRIBE calls keep coming in.
>
> Mesos authentication is completely disabled.
>
>
> What could be possible causes for this problem?
>
>
> Thanks,
>
> Sebastian
>
>
>
>
> ------------------------------------------------------------------------
> Disclaimer The information in this email and any attachments may contain
> proprietary and confidential information that is intended for the
> addressee(s) only. If you are not the intended recipient, you are hereby
> notified that any disclosure, copying, distribution, retention or use of
> the contents of this information is prohibited. When addressed to our
> clients or vendors, any information contained in this e-mail or any
> attachments is subject to the terms and conditions in any governing
> contract. If you have received this e-mail in error, please immediately
> contact the sender and delete the e-mail.
>