You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@slider.apache.org by Gour Saha <gs...@hortonworks.com> on 2015/06/15 18:26:44 UTC

Re: Accessing the application export information from ZooKeeper

Apologies for missing this completely.

ZK is not a good place to publish a lot of data (specifically dynamic data). I think that's one of reason why only the REST endpoint is published to ZK which is then used to fetch the export data.

There are discussions going on in the YARN world to create hooks to update DNS dynamically based on registry. That would be the most reliable way, I guess.

https://issues.apache.org/jira/browse/YARN-913

-Gour


From: Jatin Kumar <jk...@rocketfuelinc.com>>
Date: Tuesday, June 9, 2015 at 1:20 AM
To: Gour Saha <gs...@hortonworks.com>>
Cc: dev <de...@slider.incubator.apache.org>>
Subject: Accessing the application export information from ZooKeeper

Hello Gour & Team,

Thanks for giving a good tool to exploit Yarn resources!

We know that Slider publishes AM RPC details to ZK which can then be used to fetch the export information directly from AM over RPC but I am not very sure if there is any piece of configuration which can be used to make slider publish the export information (ip:port of user app) to ZooKeeper.

If there is not a direct way then one of the many possible ways is that provider itself publishes information (ip:port) to ZK. Can you please suggest if that would be a reliable way to keep ZK up to date?

--
Thanks,
Jatin

Re: Accessing the application export information from ZooKeeper

Posted by Steve Loughran <st...@hortonworks.com>.
On 15 Jun 2015, at 17:58, Lei Guo <le...@huawei.com>> wrote:

Updating DNS dynamically may generate security concern in enterprise production environment, even Yarn has its own DNS server. We hit this trouble about 10 years ago for a Linux service management tool we built. Especially when some organization have the DNS server maintained by infrastructure team, and the service management tool is owned by application/platform team, this will be a blocking issue.

YARN doesn't have its own DNS server yet. And yes, DNS is at risk of creating conflict. Note that docker's etcd is a DNS server, and of course modern printers and any OSX laptop is running mDNS, which is a multicast DNS protocol.

The discussions I've been having with Sanjay Radia, have come around to

  1.  configure Bind, rather than trying to implement our own DNS protocol.
  2.  have the DNS server not relay unresolved addresses. It'd be for registry resolution only.
  3.  Maybe make it a pure ZK -> DNS binding, rather than just yarn-registry only. This would make it even more broadly useful (and ZK would make a nice HA DNS service)
  4.  keep it optional

FWIW many of the big Hadoop clusters all run caching DNS servers locally as it reduces load on the main infrastructure DNS servers, and removes them as a point of failure.

-steve

RE: Accessing the application export information from ZooKeeper

Posted by Lei Guo <le...@huawei.com>.
Updating DNS dynamically may generate security concern in enterprise production environment, even Yarn has its own DNS server. We hit this trouble about 10 years ago for a Linux service management tool we built. Especially when some organization have the DNS server maintained by infrastructure team, and the service management tool is owned by application/platform team, this will be a blocking issue.

Have a nice day,
Lei

华为技术有限公司 Huawei Technologies Co., Ltd. Canada Research Center



Phone: 905-944-5000 ext 7585
Email: leiguo@huawei.com
Huawei Technologies CANADA CO.., Ltd.
60 Columbia Way, Suite 102
Markham, Ontario, Canada L3R 0C9
http://www.huawei.com 
本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁
止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中
的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This e-mail and its attachments contain confidential information from HUAWEI, which 
is intended only for the person or entity whose address is listed above. Any use of the 
information contained herein in any way (including, but not limited to, total or partial 
disclosure, reproduction, or dissemination) by persons other than the intended 
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by 
phone or email immediately and delete it!

-----Original Message-----
From: Gour Saha [mailto:gsaha@hortonworks.com] 
Sent: June-15-15 12:27 PM
To: Jatin Kumar
Cc: dev
Subject: Re: Accessing the application export information from ZooKeeper

Apologies for missing this completely.

ZK is not a good place to publish a lot of data (specifically dynamic data). I think that's one of reason why only the REST endpoint is published to ZK which is then used to fetch the export data.

There are discussions going on in the YARN world to create hooks to update DNS dynamically based on registry. That would be the most reliable way, I guess.

https://issues.apache.org/jira/browse/YARN-913

-Gour


From: Jatin Kumar <jk...@rocketfuelinc.com>>
Date: Tuesday, June 9, 2015 at 1:20 AM
To: Gour Saha <gs...@hortonworks.com>>
Cc: dev <de...@slider.incubator.apache.org>>
Subject: Accessing the application export information from ZooKeeper

Hello Gour & Team,

Thanks for giving a good tool to exploit Yarn resources!

We know that Slider publishes AM RPC details to ZK which can then be used to fetch the export information directly from AM over RPC but I am not very sure if there is any piece of configuration which can be used to make slider publish the export information (ip:port of user app) to ZooKeeper.

If there is not a direct way then one of the many possible ways is that provider itself publishes information (ip:port) to ZK. Can you please suggest if that would be a reliable way to keep ZK up to date?

--
Thanks,
Jatin