Fine-grained mesos execution hangs on Debian 7.4

[Fedechicco <fe...@gmail.com>]

Hello,

as from object, when I run scala spark-shell on our mesos (0.19) cluster
some spark slaves just hang at the end of the staging phase for any given
elaboration.

The cluster has mixed OSes (Ubuntu 14.04 / Debian 7.4), but if I run the
same shell and commands using coarse grained mode everything works just
fine.

I'm using a spark 1.1.0-SNAPSHOT built from sources (pulled today from
git), on openjdk-7.

Sadly I can't get any error message from the sandboxes for the hanging
slaves, everything seems in order, just stuck.

Any suggestion on how to debug this?

thanks,
Federico

Re: Fine-grained mesos execution hangs on Debian 7.4

[Fedechicco <fe...@gmail.com>]

Hello Sebastien,

it is not working with the 1.0 branch either.
I decided to compile spark from source precisely because of the
[SPARK-2204] fix, because before that I couldn't get fine-grained working
at all.
Now it works fine if the cluster is only composed of Ubuntu 14.04 nodes,
and when I introduce the Debian 7.4 nodes they hang like described.

I tested both the master (cloned with the [SPARK-2204] fix already inside)
and the 1.0 branch with that commit cherry-picked inside.
The behaviour is the same: whatever the reason is, it has not been
introduced after the 1.0 release.

Did anybody else test fine-grained with a Debian 7 or 7.4?




2014-06-26 19:23 GMT+00:00 Sébastien Rainville <sebastienrainville@gmail.com
>:

> Hello Federico,
>
> is it working with the 1.0 branch? In either branch, make sure that you
> have this commit:
> https://github.com/apache/spark/commit/1132e472eca1a00c2ce10d2f84e8f0e79a5193d3
> I never saw the behavior you are describing, but that commit is important
> if you are running in fine-grained mode, and it was merged only yesterday.
>
> - Sebastien
>
>
>
> On Thu, Jun 26, 2014 at 12:11 PM, Fedechicco <fe...@gmail.com> wrote:
>
>> Hello,
>>
>> as from object, when I run scala spark-shell on our mesos (0.19) cluster
>> some spark slaves just hang at the end of the staging phase for any given
>> elaboration.
>>
>> The cluster has mixed OSes (Ubuntu 14.04 / Debian 7.4), but if I run the
>> same shell and commands using coarse grained mode everything works just
>> fine.
>>
>> I'm using a spark 1.1.0-SNAPSHOT built from sources (pulled today from
>> git), on openjdk-7.
>>
>> Sadly I can't get any error message from the sandboxes for the hanging
>> slaves, everything seems in order, just stuck.
>>
>> Any suggestion on how to debug this?
>>
>> thanks,
>> Federico
>>
>
>

Re: Fine-grained mesos execution hangs on Debian 7.4

[Sébastien Rainville <se...@gmail.com>]

Hello Federico,

is it working with the 1.0 branch? In either branch, make sure that you
have this commit:
https://github.com/apache/spark/commit/1132e472eca1a00c2ce10d2f84e8f0e79a5193d3
I never saw the behavior you are describing, but that commit is important
if you are running in fine-grained mode, and it was merged only yesterday.

- Sebastien



On Thu, Jun 26, 2014 at 12:11 PM, Fedechicco <fe...@gmail.com> wrote:

> Hello,
>
> as from object, when I run scala spark-shell on our mesos (0.19) cluster
> some spark slaves just hang at the end of the staging phase for any given
> elaboration.
>
> The cluster has mixed OSes (Ubuntu 14.04 / Debian 7.4), but if I run the
> same shell and commands using coarse grained mode everything works just
> fine.
>
> I'm using a spark 1.1.0-SNAPSHOT built from sources (pulled today from
> git), on openjdk-7.
>
> Sadly I can't get any error message from the sandboxes for the hanging
> slaves, everything seems in order, just stuck.
>
> Any suggestion on how to debug this?
>
> thanks,
> Federico
>