You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@etch.apache.org by "scott comer (sccomer)" <sc...@cisco.com> on 2008/12/22 21:53:15 UTC
Etch Name Service Proposal
this is a proposal for a name service for etch. there is a text document
describing the
name service and rationale, and a proposed etch service idl.
the intention is to define the service interface, implement an etch
scheme which uses it,
and produce a reference implementation of the service which can support
simple
deployments and also test the etch scheme.
i will also post this to the wiki. please review and comment here.
scott out
Re: Etch Name Service Proposal
Posted by James Dixson <di...@gmail.com>.
A couple of call flow diagram would be useful.
Also a re-enforcement of how having the name service defined in terms
of an Etch API allows the service to be easily consumed by all
Etch-enabled languages.
--
james
On Mon, Dec 22, 2008 at 2:53 PM, scott comer (sccomer)
<sc...@cisco.com> wrote:
> this is a proposal for a name service for etch. there is a text document
> describing the
> name service and rationale, and a proposed etch service idl.
>
> the intention is to define the service interface, implement an etch scheme
> which uses it,
> and produce a reference implementation of the service which can support
> simple
> deployments and also test the etch scheme.
>
> i will also post this to the wiki. please review and comment here.
>
> scott out
>
>
> Introduction
>
> When an client is connecting to an etch service, a uri is used to specify
> the
> details of the connection. Typically this code looks something like this:
>
> RemoteNameServiceServer server = NameServiceHelper.newServer(
>
> "tcp://host:4001?filter=KeepAlive&TcpTransport.reconnectDelay=4000",
> null, "factory" );
>
> The uri specifies all the details of the etch connection in a convenient
> form.
> Yet the uri is still hard to manage. With it embedded in code we must
> recompile
> the client to change the connection details. We can load the uri from some
> other place, say a configuration file, environment variable, or the command
> line, but all we've done is move the problem to yet another hard to manage
> place. (Here 'manage' specifically refers to the process of obtaining the
> uri or
> updating it after a change.)
>
> What can we do to make the uri easier to manage? The main issue is that the
> service implementation and deployment specifies what the uri should be. Any
> change to the service implementation or deployment could trigger a need to
> update the uri that clients use to connect to it. So, one fact (connection
> uri),
> one source (service implementation and deployment), but a delivery path
> which is
> not automatic and often includes a human. Can you imagine trying to update
> 30,000 clients with a new connection uri?
>
> The classic solution applies here and represents the start of the next phase
> in
> etch development. Etch needs a Name Service.
>
> Basics
>
> At its heart a name service is easy. We use them every day without really
> thinking about them. Given a simple but abstract identifier a more
> complicated
> but concrete identifier is produced by some sort of lookup. In exchange for
> the
> abstraction, we obtain some independence from the complicated details of the
> concrete identifier, allowing the concrete identifier to change as needed by
> the environment. For example, apache.org is translated into the internet
> protocol v4 address 140.211.11.130 for us by dns. My name, sccomer, is
> translated into a uid (1079) by my nearby linux box and used for all sorts
> of
> evil purposes. In both cases I can use the simple name as a substitute for
> the
> more complicated name seamlessly in the environments I work in. Let's call
> the
> abstract identifier the source. Let's call the concrete identifier the
> target.
>
> In order for the name service to be useful to me, I have to be able to
> depend
> upon and trust it. That is, the translation process needs to be available
> when
> I need it, and the translations need to be accurate and, most importantly,
> secure. By secure I mean that if I'm going to connect to a service and
> supply
> some credentials and use it to do my work, I have an expectation that the
> service itself is reliable and trustworthy. That it is not being spoofed.
> This
> translates into access controls limiting who can make changes to the
> underlying
> data used to implement the name service translation.
>
> Requirements
>
> The name service should be defined using etch. Perhaps this is obvious but
> I'll
> say it here for completeness.
>
> The source should be specified in a uri. This allows us to exchange the
> target
> uri for a source uri + name service api to achieve our goal.
>
> To implement the functionality of the name service we need to have these
> elements:
>
> 1) an api to access the name translations, update them, etc.
> 2) a mechanism to protect the name service api from unauthorized access.
> 3) an etch connection scheme which uses the name service api to automate the
> translation process for the client.
>
> First the api elements:
>
> 1a) format for source.
> 1b) a method to translate source to target.
> 1c) a method to list translations matching a query string.
> 1d) a method to add translation of a source.
> 1e) a method to remove translation of a source.
> 1f) a method to associate additional qualities with a particular
> translation,
> which might be used to further qualify the translation for a particular
> purpose.
>
> In order to implement replicated name services, we need an ability to
> receive
> notification of changes to translations at a name service instance:
>
> 1f) a method to subscribe to changes.
> 1g) a method to unsubscribe to changes.
> 1h) a method to receive notification of changes.
> 1i) a method to differentiate translations from primary or secondary
> sources.
>
> These mechanisms protect access:
>
> 2a) methods that read need to check authorization for reading (1b, 1c, 1f).
> 2b) methods that update need to check authorization for updating (1d, 1e).
>
> Etch client access to name service:
>
> 3a) Etch clients use a uri to specify etch connection parameters, so it is
> natural that the way to specify a source requiring name service
> translation
> would be with a uri which uses an alternate scheme along with the
> requested
> source. Some elements of this source uri must be copied over to the
> resulting target uri after translation, perhaps overriding elements
> there.
>
> Source Format
>
> Within a given name service database may be many entries offering the same
> (or, essentially the same) api, that is, etch service name (e.g.,
> etch.examples.perf.Perf). This is just as it is for any other service
> available over the network (nfs file servers, smtp mail servers, jabber im
> servers, etc.). This suggests that the api might be useful as an
> organizational
> concept for the name service database.
>
> Suppose we partition the name space into domains based on the api being
> offered.
> Within a given api domain, there may be a number of named instances. The
> instance name is used to uniquely identify a running instance of the service
> offering the api. There may be several ways to access an instance (called
> schemes in etch). The instance might offer tls and soap schemes, for
> example.
> Finally, a given scheme may be available in more than one flavor, depending
> upon the capabilities of the client. The preferred scheme should be tried
> first,
> and if it isn't supported a second scheme is tried, third, etc.
>
> This suggests a four part name: api, instance, scheme, and priority. We can
> combine the four parts into a single name by using the slash character as a
> separator. This gives the name a path-like quality and also allows it to be
> easily embedded in a uri:
>
> api/instance/scheme/priority
>
> One obvious way to express api is to use the fully qualified service name of
> the
> etch idl. These names are composed of standard identifiers separated by
> periods
> (e.g., etch.examples.perf.Perf).
>
> The instance name should not contain the slash character. Since we want to
> embed
> these in a uri, the instance name should not contain any other uri
> significant
> characters either. If we stick to the same format as the api, we still have
> a
> large and interesting name space to work with.
>
> Since the scheme corresponds to a uri scheme name, then the same uri scheme
> syntax is required. This is pretty much a standard identifier.
>
> Priority is most easily a positive non-zero integer, with 1 being highest
> priority.
>
> So, here is a fully specified source for the Perf service named foo with tcp
> scheme:
>
> etch.examples.perf.Perf/foo/tcp/1
>
> Since the sequence number is not often needed, it could be defaulted to 1 if
> missing. This would be the same as the previous:
>
> etch.examples.perf.Perf/foo/tcp
>
> Name Service Api Details
>
> Please see the accompanying ns.etch file for specific api details and
> documentation.
>
> Etch Scheme
>
> An etch scheme is introduced to gain automatic access to the name service.
> This
> removes from most clients any burden relating to name service, and makes
> using
> name server essentially transparent (i.e., no program changes are required).
>
> Here's an example etch scheme:
>
> etch://etch.examples.perf.Perf/foo/tcp
>
> This would connect to a name service, lookup the source, and then connect to
> the returned target. The highest priority entry matching the other three
> parameters would be used.
>
> Here is another example, this time with modification to the target uri:
>
> etch://etch.examples.perf.Perf/foo/tcp?add:filter=Logger
>
> Logger would be added to the target filter chain.
>
> Less specific source specifications might be used to request alternate
> capabilities. Here we request a connection, tls would be preferred but we
> will
> accept tcp:
>
> etch://etch.examples.perf.Perf/foo/tls,tcp
>
> Here we request the nearest service to 78749 within 100 miles:
>
> etch://etch.examples.perf.Perf/./tcp?distance_le=100mi&zip=78749
>
> (to implement this, the name service would need to have location of
> services and some sort of function calculating distance based on zip codes.)
>
> Q/A:
>
> What other target uri edits can you make?
>
> This is all kinda twitchy and not well-baked. Speculation. The
> problem is,
> without edits there is clearly a gap in capability. But edits are
> messy
> and too many puts us back to first base.
>
> set:option=value (adds/replaces the query term option=value)
> del:option (deletes the query term option)
> rem:option=value (removes value from option's comma separated list)
>
> Which name service is used?
>
> The name service might be configured in a number of ways:
> Environment varible
> Via resources
> Dhcp option (configures all clients within a subnet)
>
> Can name service be partitioned or federated?
>
> Yes. The instance could have structure which suggests a hierarchy of
> name
> services serving various domains. A request to lookup or register an
> instance not in the local domain could be routed to another name
> service for
> processing.
>
> Can name service be replicated?
>
> Yes. Using the subscribe feature, one name service could replicate
> the
> contents of another (in a publisher/subscriber fashion). Lookup could
> be
> satisfied locally, while register could flow through to the publisher
> or
> the relationship could be more symmetric.
>
> If I connect to a service using the uri "etch://Foo/bar/tcp" and the
> connection
> goes down, how do I reconnect?
>
> This is tricky. I don't know yet. Originally I was thinking that the
> name
> would be looked up and then the connection made to the first target
> which
> worked. That is, the actual transport stack would reflect the matched
> target.
> But if the target uri for that name changes, or a secondary
> translation is
> used, then the bottom layers of the current stack would have to be
> discarded and rebuilt from the new target uri.
>
> module etch.services.ns
>
> /**
> * The NameService provides translation from an abstract name for a service
> to a
> * uri that may be used to contact the service. A given service may offer
> * several different connection schemes, and a client may only support a
> subset
> * of those. So, when a client wishes to connect to a service, it may query
> * the NameService with both abstract name and desired schemes in order to
> find
> * a suitable match.
> *
> * Some example queries:
> *
> * All services with servicename='Foo' and instancename='bar' and
> scheme='tcp'
> *
> * serviceUri='Foo/bar/tcp'
> *
> * Same as the query above, but written out longhand:
> *
> * servicename='Foo' and instancename='bar' and scheme='tcp'
> *
> * Service Foo instance bar with tls or tcp (prefer tls):
> *
> * serviceUri='Foo/bar/tls,tcp'
> *
> * or:
> *
> * servicename='Foo' and instancename='bar' and scheme='tls,tcp'
> *
> * Fully qualified serviceUri:
> *
> * serviceUri='Foo/bar/tcp/1'
> *
> * Any instance of service Foo located in Austin, TX:
> *
> * servicename='Foo' and qualities.location='Austin, TX'
> *
> * Instances of service Foo with capacity >= 400:
> *
> * servicename='Foo' and qualities.capacity>=400
> *
> * Relative operations on qualities require that the quality be present and
> be
> * comparable (with mixed types some type promotion is supported). But you
> might
> * be surprised, qualities which are absent will give a negative comparison
> no
> * matter which sense is used: both qualities.x > 50 and qualities.x < 50
> are
> * false whenever qualities.x is null. Consider this test: not qualities.x <
> 50.
> * This is not the same as qualities.x >= 50. It is the same as qualities.x
> =
> * null or qualities.x >= 50.
> */
> @Timeout( 30000 )
> service NameService
> {
> /** An entry describing a service. */
> struct Entry
> (
> /** A service description uri, composed of
> * servicename/instancename/scheme[/priority]. */
> string serviceUri,
>
> /** Qualities of this service instance. */
> Map qualities,
>
> /** Etch service connection uri, for example
> * tcp://localhost:9000?filter=KeepAlive */
> string targetUri,
>
> /** Lifetime in seconds from last update. */
> int ttl,
>
> /** Who created or last update. */
> string who,
>
> /** Date / time of create or last update. */
> Datetime lastUpdate,
>
> /** Flag indicating entry has been removed. */
> boolean removed
> )
>
> /**
> *
> */
> @Authorize( canLookup, source )
> Entry lookup( string source )
>
> /**
> * Looks up entries by matching them against the query string.
> Entries are
> * returned in a stable and consistent order, increasing alphabetical
> by
> * servicename, instancename, and scheme, and then increasing
> numerical by
> * priority. If a scheme search term appears with a list, schemes are
> * returned in the specified order (e.g., scheme='tls,tcp').
> *
> * @param query a sql-like query expression using the elements of the
> * serviceUri, targetUri, and qualities. If query is null or the
> empty
> * string, all entries are matched.
> *
> * @param offset offset in the list of matched items of the first
> item to
> * return. This parameter and the count parameter are used to step
> through
> * the result set when there are many results.
> *
> * @param count number of items to return.
> *
> * @return entries matching query. If you requested 10 items and 10
> are
> * returned, there could be more. To get the next batch, add
> results.length
> * to offset and call find again:
> *
> * int i = 0;
> * int n = 10;
> * Entry[] results;
> * while ((results = service.find( null, i, n )).length > 0)
> * {
> * for (Entry e: results)
> * processEntry( e );
> * if (results.length < n)
> * break;
> * i += results.length;
> * }
> */
> @Authorize( canFind, query )
> Entry[] find( string query, int offset, int count )
>
> /**
> * Adds or updates the specified entry. The given parameters replace
> any
> * existing values in an Entry whose key is serviceUri, whereas who
> and
> * lastUpdate are set to the current user and current date / time,
> * respectively.
> *
> * @param serviceUri the uri describing the service. The uri should
> be of
> * the form "servicename/instancename/scheme[/priority]", where
> servicename
> * is a valid fully qualified service name (e.g.,
> * etch.services.ns.NameService"), instancename is a valid etch
> identifier
> * (e.g, fred, alice01), scheme is a valid uri scheme (e.g., tcp,
> tls), and
> * priority is an integer >= 1. If priority is omitted, it is
> defaulted to
> * 1.
> *
> * @param qualities a map which may be used to describe additional
> features
> * of the entry, such as purpose, licenses, capacity, location,
> owner,
> * whatever. Query strings may test values of qualities using a
> variety of
> * sql-like operators. Qualities may be null.
> *
> * @param targetUri the uri describing the contact information for
> the
> * service.
> *
> * @param ttl the lifetime of the entry specified as seconds. 0 means
> * forever, -1 means remove immediately when the connection to the
> * NameService is dropped.
> */
> @Authorize( canRegister, serviceUri, qualities )
> void register( string serviceUri, Map qualities, string targetUri,
> int ttl )
>
> /**
> * Registers a number of entries all in one operation. Identical to
> calling
> * register with each entry in turn.
> *
> * @param entries a sequence of Entry records with serviceUri,
> targetUri,
> * qualities, and ttl as specified in register() above. Who and
> lastUpdate
> * fields are ignored.
> */
> @Authorize( canRegisterBulk, entries )
> void registerBulk( Entry[] entries )
>
> /**
> * Removes the specified entry.
> *
> * @param serviceUri the uri describing the service.
> */
> @Authorize( canRegister, serviceUri, null )
> void unregister( string serviceUri )
>
> /**
> * Adds a request for notification of changes to entries matching the
> query.
> * The current value of all matching entries is delivered via
> entryChanged
> * client message, as well as any updates or new entries.
> *
> * @param query a sql-like query expression using the elements of the
> * serviceUri, targetUri, and qualities. If query is null or the
> empty
> * string, all entries are matched.
> */
> @Authorize( canFind, query )
> void subscribe( string query )
>
> /**
> * Removes a request for notification of changes to entries matching
> the
> * query.
> *
> * @param query a query previously passed to subscribe. The string
> must
> * match exactly.
> */
> void unsubscribe( string query )
>
> /**
> * Removes all requests for notification of changes to entries. This
> * operation is implicitly performed when the connection to the
> NameService
> * is dropped.
> */
> void unsubscribeAll()
>
> ////////////////////
> // RIGHTS TESTING //
> ////////////////////
>
> /**
> * Tests whether the current user is authorized to lookup the source.
> * @param source the complete specification api/instance/scheme.
> */
> boolean canLookup( string source )
>
> /**
> * Tests whether the current user is authorized to run the query.
> *
> * @param query a sql-like query expression using the elements of the
> * serviceUri, targetUri, and qualities. If query is null or the
> empty
> * string, all entries are matched.
> *
> * @return true if the current user is authorized to run the query.
> */
> boolean canFind( string query )
>
> /**
> * Tests whether the current user is authorized to register the
> service.
> *
> * @param serviceUri the uri describing the service.
> *
> * @param qualities a map which may be used to describe additional
> features
> * of the entry.
> *
> * @return true if the current user is authorized to register the
> service.
> */
> boolean canRegister( string serviceUri, Map qualities )
>
> /**
> * Tests whether the current user is authorized to register the
> entries.
> * This is the same as:
> *
> * for (Entry entry: entries)
> * if (!canRegister( entry.serviceUri, entry.qualities ))
> * return false;
> * return true;
> *
> * @param entries a sequence of Entry records with serviceUri,
> targetUri,
> * qualities, and ttl as specified in register() above. Who and
> lastUpdate
> * fields are ignored.
> *
> * @return true if the current user is authorized to register the
> entries.
> * This is an all or nothing proposition.
> */
> boolean canRegisterBulk( Entry[] entries )
>
> ////////////////////
> // CLIENT METHODS //
> ////////////////////
>
> /**
> * Notifies the client of a change in an entry. The entry might have
> been
> * created, updated, or removed.
> *
> * Note: while this might have normally been an event, we made it a
> call
> * to slow down the processing of what might otherwise be a rather
> large
> * change set.
> *
> * Note: when keeping track of entries, always keep the one with the
> latest
> * lastUpdate.
> *
> * @param query the query which triggered the notification.
> *
> * @param entry the entry which has changed.
> */
> @Direction( client )
> void entryChanged( string query, Entry entry )
> }
>
>
RE: Etch Name Service Proposal
Posted by "J.D. Liau (jliau)" <jl...@cisco.com>.
Looks great, should be vary helpful to large deployment.
-----Original Message-----
From: Scott Comer (sccomer)
Sent: Monday, December 22, 2008 2:53 PM
To: etch-dev@incubator.apache.org
Subject: Etch Name Service Proposal
this is a proposal for a name service for etch. there is a text document
describing the name service and rationale, and a proposed etch service
idl.
the intention is to define the service interface, implement an etch
scheme which uses it, and produce a reference implementation of the
service which can support simple deployments and also test the etch
scheme.
i will also post this to the wiki. please review and comment here.
scott out
Re: Etch Name Service Proposal
Posted by Niclas Hedhman <ni...@hedhman.org>.
On Tue, Jan 6, 2009 at 11:07 PM, James Dixson <di...@gmail.com> wrote:
> No. We have not really looked into Apache River/Jini as a possible
> solution, nor have we looked at Zeroconf or UPnP.
I was not requesting you to look at any of these as possible
solutions, but the Jini specs contain a lot of useful contents in this
space which you could learn from and the adapt into the Etch
specifications. Examples;
* Leases, a general-purpose mechanism for timing out resource
allocation on an intermittent network.
* Discovery/Join, a mechanism to locate the Naming Service (called
Lookup Service in Jini). The entire cycle of Discover -> Join -> Get
Hold Of Name Service -> Register Service (for the service PoV) and
likewise the Discover -> Join -> NameService -> Lookup Service for the
clients, are some things I think Etch should incorporate.
* Remote Events, tackles aspects of events on networks. You probably
have that well covered...
IMHO, Jini have a lot of good ideas and well-written specifications,
but that the implementations have been 'hard to consume' and the
community been 'defensive' to improve. My hope was that Etch can pick
up the stick (the good ideas) and bring it to the masses.
Cheers
Niclas
--
http://www.qi4j.org - New Energy for Java
Re: Etch Name Service Proposal
Posted by James Dixson <di...@gmail.com>.
No. We have not really looked into Apache River/Jini as a possible
solution, nor have we looked at Zeroconf or UPnP. Our initial effort
has been to define the Etch Name Service in terms of Etch itself. In
other words, to define the service in terms of its Etch API and
behaviors rather than its specific implementation. The purpose of
defining the service in terms of Etch is to maintain our platform and
language neutrality goals and to keep the developer experience at the
level of an API call.
Apache River may actually be a good 90% implementation of the behavior
we are seeking here. If that is the case then we could implement the
Etch API for the Etch Name Service on top of Jini and we would be
good. I just do not know enough about Apache River to evaluate how
realistic/practical that approach might be. It is certainly worth
investigating if only as you say to learn from their experiences.
--
james
On Tue, Dec 23, 2008 at 1:02 AM, Niclas Hedhman <ni...@hedhman.org> wrote:
> On Tue, Dec 23, 2008 at 4:53 AM, scott comer (sccomer)
> <sc...@cisco.com> wrote:
>> this is a proposal for a name service for etch. there is a text document
>> describing the
>> name service and rationale, and a proposed etch service idl.
>>
>> the intention is to define the service interface, implement an etch scheme
>> which uses it,
>> and produce a reference implementation of the service which can support
>> simple
>> deployments and also test the etch scheme.
>
> How much have you guys looked at Jini (now Apache River) before? It
> addresses these same concerns, possibly in other scope than relevant
> here, but there are a lot of lessons learned from a now 10 year old
> technology.
>
> Formal specs are available at;
> http://www.jini.org/wiki/Category:Jini_Specifications
>
>
> Cheers
> Niclas
>
Re: Etch Name Service Proposal
Posted by Niclas Hedhman <ni...@hedhman.org>.
On Tue, Dec 23, 2008 at 4:53 AM, scott comer (sccomer)
<sc...@cisco.com> wrote:
> this is a proposal for a name service for etch. there is a text document
> describing the
> name service and rationale, and a proposed etch service idl.
>
> the intention is to define the service interface, implement an etch scheme
> which uses it,
> and produce a reference implementation of the service which can support
> simple
> deployments and also test the etch scheme.
How much have you guys looked at Jini (now Apache River) before? It
addresses these same concerns, possibly in other scope than relevant
here, but there are a lot of lessons learned from a now 10 year old
technology.
Formal specs are available at;
http://www.jini.org/wiki/Category:Jini_Specifications
Cheers
Niclas
RE: Etch Name Service Proposal
Posted by "Gaurav Sandhir (gsandhir)" <gs...@cisco.com>.
Looks good.
-Gaurav
-----Original Message-----
From: Scott Comer (sccomer)
Sent: Monday, December 22, 2008 2:53 PM
To: etch-dev@incubator.apache.org
Subject: Etch Name Service Proposal
this is a proposal for a name service for etch. there is a text document
describing the name service and rationale, and a proposed etch service
idl.
the intention is to define the service interface, implement an etch
scheme which uses it, and produce a reference implementation of the
service which can support simple deployments and also test the etch
scheme.
i will also post this to the wiki. please review and comment here.
scott out