registry use cases

[Steve Loughran <st...@hortonworks.com>]

Hi,

I'm starting to build a service registry in the now-in-incubation Slider
project, with an ultimate goal for a derivative or rework of this to make
its way into YARN proper, so that there'll be one we can all use to publish
and share information about YARN-deployed and static apps in a Hadoop
cluster.

To this end I'm trying to make sure I understand the various uses other
projects are making of their ZK-based registries.

I see that Twill is using it to publish (hostname, port) pairs, and lets
you poll or watch for this, and want to know a bit more


   1. What are the common ways you use this? Is it within a distributed
   app, or between apps?
   2. Do you ever use this outside the cluster itself?
   3. Do you listen for changes, or just grab it at startup?
   4. If you could publish more information, what would you publish?
   5. If we had a well-known URL a service registry, one that would support
   GET lookups and 302 redirects, what would you serve up? A Web UI for
   people, a REST API for app management, core app functionality...
   6. What security model do you rely on? do you assume that whoever
   publishes a service is being honest, do you rely on ZK permissions, or
   something else?
   7. Finally: did you deliberately choose not to use Apache Curator's
   discovery mechanism, and if so -why not?


Any insight to these or other registry-issues would be really helpful. I
don't want to build the "single best ever registry service for humanity" as
it would take too long, but I'd like to start with something that we can
evolve, which suits YARN app needs and hooks into the user experience and
management tools. For now we're using Curator, with some workarounds for
hadoop 2.4 jackson versions, a custom payload that publishes the real info,
and our AM serving up content -configurations and other artifacts- indexed
off the registry entry.


-Steve

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: registry use cases

[Steve Loughran <st...@hortonworks.com>]

On 16 May 2014 08:12, Andreas Neumann <an...@apache.org> wrote:

> Hi Steve,
>
> not sure how long ago you posted this, I still get emails that were sent
> during the outage, so apologies if this response comes with a little delay.
>

yes, it did get lost, but thanks fore replying


> Let me start with answers to your list of questions (In some cases, this
> reflects the way that Twill is used at Continuuity, there may be others who
> use it differently):
>
>    1. What are the common ways you use this? Is it within a distributed
> >    app, or between apps?
>
>
> We use it within an app (one runnable discovers the location of another
> runnable in the same app) and between different Twill apps. We also use it
> to discover services provided by a Twill app from outside of the cluster.
>
>
OK. Helix has the notion of internal vs external here (a bit like
private/public, but json introspection doesn't like those words)


>    2. Do you ever use this outside the cluster itself?
>
>
> Yes, we also use it outside of the cluster. Twill's registry is not limited
> to use within Twill.
>
>    3. Do you listen for changes, or just grab it at startup?
>
>
> Twill's discovery service client watches for changes in ZooKeeper and
> provides live updates.
>
>    4. If you could publish more information, what would you publish?
>
>
> I think that really depends on the application. From the perspective of the
> Twill framework, it is really about the location, but I see that generic
> payloads would be of good use.
>

OK. We're playing with publishing key-value configs, but that's the kind of
thing that isn't ready for standardisation. I think for the YARN core we
will initially define

-the notion of an app having multiple components, each potentially
publishing their own data
-the notion of an "instance" entry that can represent the primary accessor
to the application (this could be registered by any component, its just the
default thing to bind to
-the notion of  internal/external
-the notion of a service/node having 1* endpoint (web, IPC, REST, ...),
with information for client apps and UIs. I think we may want to evolve
some common names here, for things like JMX, log4j management, ...
-a way for components to publish 1* JSON document of their own choice.
Anything can do their own, or build defacto standards with others.





>
>    5. If we had a well-known URL a service registry, one that would support
> >    GET lookups and 302 redirects, what would you serve up? A Web UI for
> >    people, a REST API for app management, core app functionality...
>
>
> Not sure if REST is the right protocol for a service discovery, as you
> cannot watch for changes, you would have to keep polling, which seems
> inappropriate to me. WebSockets might be better for this. 302 redirects
> also assume that the registered services are all HTTP, which may not be the
> case.
>
>    6. What security model do you rely on? do you assume that whoever
> >    publishes a service is being honest, do you rely on ZK permissions, or
> >    something else?
>
>
> Security is not quite evolved in Twill yet. We currently rely on ZK
> permissions.
>
>    7. Finally: did you deliberately choose not to use Apache Curator's
> >    discovery mechanism, and if so -why not?
>
>
> This was deliberate... mostly to avoid the transitive dependencies that
> Curator would bring in (and version conflicts with other libraries we use).
>

yeah, we get that. I patched Hadoop to at least use the same jackson
version,


> We also found Curator's logging very chatty and flooding our logs whereas
> it was not very resilient to ZK timeout or connection loss. However, that
> was with a Curator version from one or two years ago, and I cannot say
> whether we would make same decision today as Curator has become more
> mature.
>

They have some good plugin retry policy handlers, but I haven't tested
failure.

Logs are an emergent problem in long-lived YARN apps


>
> Any insight to these or other registry-issues would be really helpful. I
> > don't want to build the "single best ever registry service for humanity"
> as
> > it would take too long, but I'd like to start with something that we can
> > evolve, which suits YARN app needs and hooks into the user experience and
> > management tools. For now we're using Curator, with some workarounds for
> > hadoop 2.4 jackson versions, a custom payload that publishes the real
> info,
> > and our AM serving up content -configurations and other artifacts-
> indexed
> > off the registry entry.
>
>
> In Twill, it the runnable itself that publishes to the registry, and not
> the AM. Doing it directly from the runnable ensures that the (ephemeral)
> node in ZK disappears right away the runnable dies that actually provides
> the service. Doing it in the AM bears the risk that the AM's view of what's
> live and what's not may be out of date, and the AM also can't make use of
> ephemeral nodes because the AM would remain alive even when a runnable
> dies.
>

makes sense. Anything published by a component that is transient to that
component instance/container should be owned by that component -else you
are missing all the watchability features.


>
> Generally speaking, I think that having a service registry in YARN would be
> useful. Yet I am not sure whether Twill would use that: Twill's APIs are
> generic and do not have dependencies on YARN. Right now, the only
> implementation is on top of YARN, but we might as well run in multiple
> threads of a single JVM or in Mesos. In these cases we would not want to
> depend on a YARN registry.
>
> -Andreas.
>
>
I think what makes sense there is making sure that the YARN one is >= your
needs, and just use it a back end. If we annotate service endpoints with
binding info (protocol, URL, auth types, certs &c) it'd be good if we could
say "must" for some of these, so it would be nice if at least at
registration time Twill could state these things -you could just discard
that info when bonding to back ends that don't cache it.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: registry use cases

[Andreas Neumann <an...@apache.org>]

Hi Steve,

not sure how long ago you posted this, I still get emails that were sent
during the outage, so apologies if this response comes with a little delay.
Let me start with answers to your list of questions (In some cases, this
reflects the way that Twill is used at Continuuity, there may be others who
use it differently):

   1. What are the common ways you use this? Is it within a distributed
>    app, or between apps?


We use it within an app (one runnable discovers the location of another
runnable in the same app) and between different Twill apps. We also use it
to discover services provided by a Twill app from outside of the cluster.

   2. Do you ever use this outside the cluster itself?


Yes, we also use it outside of the cluster. Twill's registry is not limited
to use within Twill.

   3. Do you listen for changes, or just grab it at startup?


Twill's discovery service client watches for changes in ZooKeeper and
provides live updates.

   4. If you could publish more information, what would you publish?


I think that really depends on the application. From the perspective of the
Twill framework, it is really about the location, but I see that generic
payloads would be of good use.

   5. If we had a well-known URL a service registry, one that would support
>    GET lookups and 302 redirects, what would you serve up? A Web UI for
>    people, a REST API for app management, core app functionality...


Not sure if REST is the right protocol for a service discovery, as you
cannot watch for changes, you would have to keep polling, which seems
inappropriate to me. WebSockets might be better for this. 302 redirects
also assume that the registered services are all HTTP, which may not be the
case.

   6. What security model do you rely on? do you assume that whoever
>    publishes a service is being honest, do you rely on ZK permissions, or
>    something else?


Security is not quite evolved in Twill yet. We currently rely on ZK
permissions.

   7. Finally: did you deliberately choose not to use Apache Curator's
>    discovery mechanism, and if so -why not?


This was deliberate... mostly to avoid the transitive dependencies that
Curator would bring in (and version conflicts with other libraries we use).
We also found Curator's logging very chatty and flooding our logs whereas
it was not very resilient to ZK timeout or connection loss. However, that
was with a Curator version from one or two years ago, and I cannot say
whether we would make same decision today as Curator has become more
mature.

Any insight to these or other registry-issues would be really helpful. I
> don't want to build the "single best ever registry service for humanity" as
> it would take too long, but I'd like to start with something that we can
> evolve, which suits YARN app needs and hooks into the user experience and
> management tools. For now we're using Curator, with some workarounds for
> hadoop 2.4 jackson versions, a custom payload that publishes the real info,
> and our AM serving up content -configurations and other artifacts- indexed
> off the registry entry.


In Twill, it the runnable itself that publishes to the registry, and not
the AM. Doing it directly from the runnable ensures that the (ephemeral)
node in ZK disappears right away the runnable dies that actually provides
the service. Doing it in the AM bears the risk that the AM's view of what's
live and what's not may be out of date, and the AM also can't make use of
ephemeral nodes because the AM would remain alive even when a runnable
dies.

Generally speaking, I think that having a service registry in YARN would be
useful. Yet I am not sure whether Twill would use that: Twill's APIs are
generic and do not have dependencies on YARN. Right now, the only
implementation is on top of YARN, but we might as well run in multiple
threads of a single JVM or in Mesos. In these cases we would not want to
depend on a YARN registry.

-Andreas.


On Wed, May 7, 2014 at 10:07 AM, Steve Loughran <st...@hortonworks.com>wrote:

> Hi,
>
> I'm starting to build a service registry in the now-in-incubation Slider
> project, with an ultimate goal for a derivative or rework of this to make
> its way into YARN proper, so that there'll be one we can all use to publish
> and share information about YARN-deployed and static apps in a Hadoop
> cluster.
>
> To this end I'm trying to make sure I understand the various uses other
> projects are making of their ZK-based registries.
>
> I see that Twill is using it to publish (hostname, port) pairs, and lets
> you poll or watch for this, and want to know a bit more
>
>
>    1. What are the common ways you use this? Is it within a distributed
>    app, or between apps?
>    2. Do you ever use this outside the cluster itself?
>    3. Do you listen for changes, or just grab it at startup?
>    4. If you could publish more information, what would you publish?
>    5. If we had a well-known URL a service registry, one that would support
>    GET lookups and 302 redirects, what would you serve up? A Web UI for
>    people, a REST API for app management, core app functionality...
>    6. What security model do you rely on? do you assume that whoever
>    publishes a service is being honest, do you rely on ZK permissions, or
>    something else?
>    7. Finally: did you deliberately choose not to use Apache Curator's
>    discovery mechanism, and if so -why not?
>
>
> Any insight to these or other registry-issues would be really helpful. I
> don't want to build the "single best ever registry service for humanity" as
> it would take too long, but I'd like to start with something that we can
> evolve, which suits YARN app needs and hooks into the user experience and
> management tools. For now we're using Curator, with some workarounds for
> hadoop 2.4 jackson versions, a custom payload that publishes the real info,
> and our AM serving up content -configurations and other artifacts- indexed
> off the registry entry.
>
>
> -Steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: registry use cases

[Steve Loughran <st...@hortonworks.com>]

On 16 May 2014 19:21, Terence Yim <ch...@gmail.com> wrote:

> >    5. If we had a well-known URL a service registry, one that would
> support
> >    GET lookups and 302 redirects, what would you serve up? A Web UI for
> >    people, a REST API for app management, core app functionality...
> Do you mean a hosted web service for discovery? It would be useful for
> simple use cases.
> However, the clients would need to poll for changes?
>

For any client with access to the ZK nodes, ZK makes sense. But if we have
it hosted in/near the RM, we can serve up a simple API for navigation. It
works well for debugging, but wouldn't be the right approach for the
in-YARN-cluster apps

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: registry use cases

[Terence Yim <ch...@gmail.com>]

Hi Steve,

Please see my answers inline.

On Wed, May 7, 2014 at 1:07 AM, Steve Loughran <st...@hortonworks.com> wrote:
> Hi,
>
> I'm starting to build a service registry in the now-in-incubation Slider
> project, with an ultimate goal for a derivative or rework of this to make
> its way into YARN proper, so that there'll be one we can all use to publish
> and share information about YARN-deployed and static apps in a Hadoop
> cluster.
>
> To this end I'm trying to make sure I understand the various uses other
> projects are making of their ZK-based registries.
>
> I see that Twill is using it to publish (hostname, port) pairs, and lets
> you poll or watch for this, and want to know a bit more
>
>
>    1. What are the common ways you use this? Is it within a distributed
>    app, or between apps?
Could be both. E.g. one could starts an app inside a cluster, hosting
some network endpoints
and with those endpoints announced through discovery service. Any
other services, whether
they are run inside YARN or not could discovery those endpoints and be
able to talk to them.

>    2. Do you ever use this outside the cluster itself?
Yes

>    3. Do you listen for changes, or just grab it at startup?
It's even base with client side cache updated by ZK watchers

>    4. If you could publish more information, what would you publish?
I would say just a byte array

>    5. If we had a well-known URL a service registry, one that would support
>    GET lookups and 302 redirects, what would you serve up? A Web UI for
>    people, a REST API for app management, core app functionality...
Do you mean a hosted web service for discovery? It would be useful for
simple use cases.
However, the clients would need to poll for changes?

>    6. What security model do you rely on? do you assume that whoever
>    publishes a service is being honest, do you rely on ZK permissions, or
>    something else?
Our current solution uses ZK, hence security is also relies on ZK (auth + ACL)

>    7. Finally: did you deliberately choose not to use Apache Curator's
>    discovery mechanism, and if so -why not?
Currently there are some dependencies issue with using Curator. Also
Twill uses ZK
for things other than discovery as well.

>
>
> Any insight to these or other registry-issues would be really helpful. I
> don't want to build the "single best ever registry service for humanity" as
> it would take too long, but I'd like to start with something that we can
> evolve, which suits YARN app needs and hooks into the user experience and
> management tools. For now we're using Curator, with some workarounds for
> hadoop 2.4 jackson versions, a custom payload that publishes the real info,
> and our AM serving up content -configurations and other artifacts- indexed
> off the registry entry.
>
>
> -Steve
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.