You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hama.apache.org by Diot Sebastien <s....@eurodata.de> on 2012/09/22 15:36:58 UTC
Is the sync really global?
Hi.
What happens if one job "get stuck", and doesn't sync anymore? Does the whole Hama cluster grind to an halt?
Can you have different groups of jobs that need their own independent sync cycles, but still want to communicate with other groups? Do you need to use an external API to do the communication in that case?
And are the messages really only sent over the network when a sync is reached? This would cause large spikes in network traffic. I would imagine it would be more network friendly to send the messages asap, but buffer them on the receiving end, until the sync is reached, thereby reducing the network traffic spike on a sync to only the sync signal itself.
MfG / Regards,
Sebastien Diot
Grossblittersdorferstrasse 257-259
66119 Saarbruecken
Germany
Tel.: +49 (681) 8808-0
Fax.: +49 (681) 8808-300
E-Mail.: s.diot@eurodata.de
-----------------------------------------------------
EURO DATA GmbH & Co. KG
Datenverarbeitungsdienst
HR A 6448 Amtsgericht Saarbrücken
Komplementär: A. Reiß & Sohn GmbH
HR B 4965 Amtsgericht Saarbrücken
Geschäftsführer: Dipl.-Kfm. Karl-Heinz Siebenpfeiffer
-----------------------------------------------------
Re: Is the sync really global?
Posted by Suraj Menon <su...@apache.org>.
[A Job has tasks running in parallel each contained in a peer.] Every job
has its own synchronization barrier. If one of your task is stuck on
infinite loop, it will hold up other tasks in sync. Other jobs have their
own sync barrier. Hence the whole cluster would not be blocked.
>> And are the messages really only sent over the network when a sync is
reached? This would cause large spikes in network traffic. I would imagine
it would be more network friendly to send the messages asap, but buffer
them on the receiving end, until the sync is reached, thereby reducing the
network traffic spike on a sync to only the sync signal itself.
https://issues.apache.org/jira/browse/HAMA-490 issue is for implement
asynchronous communication.
>> Can you have different groups of jobs that need their own independent
sync cycles, but still want to communicate with other groups? Do you need
to use an external API to do the communication in that case?
https://issues.apache.org/jira/browse/HAMA-639 issue would have a provision
to define tasks in this way. However, the message queue design is not
thought out for this.
-Suraj
On Sat, Sep 22, 2012 at 9:36 AM, Diot Sebastien <s....@eurodata.de> wrote:
> Hi.
>
> What happens if one job "get stuck", and doesn't sync anymore? Does the
> whole Hama cluster grind to an halt?
>
> Can you have different groups of jobs that need their own independent sync
> cycles, but still want to communicate with other groups? Do you need to use
> an external API to do the communication in that case?
>
>
>
> MfG / Regards,
> Sebastien Diot
>
> Grossblittersdorferstrasse 257-259
> 66119 Saarbruecken
> Germany
> Tel.: +49 (681) 8808-0
> Fax.: +49 (681) 8808-300
> E-Mail.: s.diot@eurodata.de
> -----------------------------------------------------
> EURO DATA GmbH & Co. KG
> Datenverarbeitungsdienst
> HR A 6448 Amtsgericht Saarbrücken
> Komplementär: A. Reiß & Sohn GmbH
> HR B 4965 Amtsgericht Saarbrücken
> Geschäftsführer: Dipl.-Kfm. Karl-Heinz Siebenpfeiffer
> -----------------------------------------------------
>
>
>