You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by ma...@orange.com on 2017/07/11 09:50:49 UTC
how to identify the alive master spark ?
Hello,
In our project, we have a Spark cluster with 2 master and 4 workers and Zookeeper decides which master is alive.
We have a problem with our reverse proxy to display the Spark Web UI. The RP redirect on a master with IP address configured in initial configuration but if Zookeeper decides to change the master, our spark Web UI is not accessible.
We search in Internet a solution to know with Zookeeper which master is alive every time but we don't find anything. It is possible with confd to search if property changed but none property is saved.In folder /spark in Zookeeper, nothing is logged.
Can you help us to solve this problem ?
Thank you
Regards,
[http://www.orange.com/sirius/logos_mail/orange_logo.gif]<http://www.orange.com/>
Marina Brunel
Ingénieur d'Etudes Confirmé
Orange Applications for Business
SCE/OAB/MTM/SM2M
Fixe +33 (0)2 99 28 88 73
Portable +33 (0)7 84 14 52 05
Interne 4873
marina.brunel@orange.com<ma...@orange.com>
3, allée de Beaulieu 35700 Rennes
www.orange-business.com/fr
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
RE: how to identify the alive master spark ?
Posted by ma...@orange.com.
Hello,
Zookeeper decides to elect another master when elected master goes down. For me, Zookeeper is always advise which is the master.
In Spark, only logs show us which is the master. A grep on logs is not a good solution for us to know the master.
For databases, Zookeeper can provide a property which indicates the hostname. Why it is not the case for Spark ?
I will see with Spark
Thanks
Regards,
Marina Brunel
-----Message d'origine-----
De : Abraham Fine [mailto:afine@apache.org]
Envoyé : mardi 11 juillet 2017 20:03
À : user@zookeeper.apache.org
Objet : Re: how to identify the alive master spark ?
Hi Marina-
This question may be better suited for the Spark mailing lists as Zookeeper doesn't really "decide" which master is alive but rather provides a mechanism for the application to make the correct decision.
Thanks,
Abe
On Tue, Jul 11, 2017, at 02:50, marina.brunel@orange.com wrote:
> Hello,
>
> In our project, we have a Spark cluster with 2 master and 4 workers and Zookeeper decides which master is alive.> We have a problem with our reverse proxy to display the Spark Web UI. The RP redirect on a master with IP address configured in initial configuration but if Zookeeper decides to change the master, our spark Web UI is not accessible.> We search in Internet a solution to know with Zookeeper which master is alive every time but we don’t find anything. It is possible with confd to search if property changed but none property is saved.In folder /spark in Zookeeper, nothing is logged.>
> Can you help us to solve this problem ?
>
> Thank you
>
> Regards,
> http://www.orange.com/sirius/logos_mail/orange_logo.gif
> *Marina Brunel*
> Ingénieur d'Etudes Confirmé
> *Orange Applications for Business*
> SCE/OAB/MTM/SM2M
>
> Fixe +33 (0)2 99 28 88 73
> Portable +33 (0)7 84 14 52 05
> Interne 4873
> marina.brunel@orange.com
>
> 3, allée de Beaulieu 35700 Rennes
> www.orange-business.com/fr
>
> _____________________________________________________________________-
> ____________________________________________________ Ce message et
> ses pieces jointes peuvent contenir des informations confidentielles
> ou privilegiees et ne doivent donc pas etre diffuses, exploites ou
> copies sans autorisation. Si vous avez recu ce message par erreur,
> veuillez le signaler a l'expediteur et le detruire ainsi que les
> pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, Orange decline toute responsabilite si ce message a ete
> altere, deforme ou falsifie. Merci. This message and its attachments
> may contain confidential or privileged information that may be
> protected by law; they should not be distributed, used or copied
> without authorisation. If you have received this email in error,
> please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have
> been modified, changed or falsified. Thank you.
>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
Re: how to identify the alive master spark ?
Posted by Abraham Fine <af...@apache.org>.
Hi Marina-
This question may be better suited for the Spark mailing lists as Zookeeper doesn't really "decide" which master is alive but rather provides a mechanism for the application to make the correct decision.
Thanks,
Abe
On Tue, Jul 11, 2017, at 02:50, marina.brunel@orange.com wrote:
> Hello,
>
> In our project, we have a Spark cluster with 2 master and 4 workers and Zookeeper decides which master is alive.> We have a problem with our reverse proxy to display the Spark Web UI. The RP redirect on a master with IP address configured in initial configuration but if Zookeeper decides to change the master, our spark Web UI is not accessible.> We search in Internet a solution to know with Zookeeper which master is alive every time but we don’t find anything. It is possible with confd to search if property changed but none property is saved.In folder /spark in Zookeeper, nothing is logged.>
> Can you help us to solve this problem ?
>
> Thank you
>
> Regards,
> http://www.orange.com/sirius/logos_mail/orange_logo.gif
> *Marina Brunel*
> Ingénieur d'Etudes Confirmé
> *Orange Applications for Business*
> SCE/OAB/MTM/SM2M
>
> Fixe +33 (0)2 99 28 88 73
> Portable +33 (0)7 84 14 52 05
> Interne 4873
> marina.brunel@orange.com
>
> 3, allée de Beaulieu 35700 Rennes
> www.orange-business.com/fr
>
> _____________________________________________________________________-
> ____________________________________________________ Ce message et
> ses pieces jointes peuvent contenir des informations confidentielles
> ou privilegiees et ne doivent donc pas etre diffuses, exploites ou
> copies sans autorisation. Si vous avez recu ce message par erreur,
> veuillez le signaler a l'expediteur et le detruire ainsi que les
> pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, Orange decline toute responsabilite si ce message a ete
> altere, deforme ou falsifie. Merci. This message and its attachments
> may contain confidential or privileged information that may be
> protected by law; they should not be distributed, used or copied
> without authorisation. If you have received this email in error,
> please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have
> been modified, changed or falsified. Thank you.
>