You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Ken Giusti <kg...@redhat.com> on 2009/11/20 19:47:19 UTC
[QMF] Proposed API for QMFv2 - RFC
Hi All,
Below is a strawman proposal for the design of the QMFv2 API. It describes an api for working with schema, the data model, and console-side operations. There's a few tbd's in it, and I still need to think about the Agent side, but I'd like to get some feedback on it early.
Would it be possible to post this as a rough draft on the QMFv2 API section of the wiki? I've tried to use confluence markup - but since I cannot submit I have no idea how (badly) it renders.
Thanks!
-K
-----------8<-------------------
h1. Proposal for "Next Generation" QMF API
h2. Goals
* Simplify QMF API
* Use new QPID Messaging API
* Implement QMF protocol via QPID Map messages
* Improve thread safety by simplifying the callback notification mechanism.
* Reorganize QMF code into common API, Console API, and Agent API modules.
h2. Common Object Classes
Object classes that are used by Agent and Console components.
h3. Schema
Schemas are used by QMF to describe the structure of management data.
h4. Schema Types
There are two classes (or types) of Schema - those that describe
QmfData objects, and those that describe QmfEvent objects.
{noformat}
SchemaTypeData:
SchemaTypeEvent:
{noformat}
h4. Schema Identifier
Schema are identified by a combination of their package name and class
name. A hash value over the body of the schema provides a revision
identifier. The class SchemaClassKey represents this Schema
identifier.
{noformat}
class SchemaClassKey:
<constructor>( package=<package-name-str>,
class=<class-name-str>,
hash=<hash-str, format="%08x-%08x-%08x-%08x">,
type=<SchemaTypeData|SchemaTypeEvent>)
.getPackageName(): returns <package-name-str>
.getClassName(): returns <class-name-str>
.getHashString(): returns <hash-str, "%08x-%08x-%08x-%08x">
.getType(): returns SchemaTypeObject or SchemaTypeEvent
{noformat}
h4. Schema for describing data Properties
The SchemaProperty class represents the description of a property
datum. Once instantiated, the SchemaProperty is immutable.
{noformat}
class SchemaProperty:
<constructor>( name=<name-value>,
type=<type-value>,
...)
.getName(): name of property (string)
.getType(): typecode for property data
.getAccess(): 1=read/create, 2=read/write, 3=read only
.getIndex(): True if this property is an index.
.isOptional(): True if this property is optional
.getUnit(): string describing units (optional)
.getMin(): minimum value (optional)
.getMax(): maximum value (optional)
.getMaxLen(): maximum length for string values (optional)
.getDesc(): optional string description of this Property
.getReference(): if type==objId, name (str) of referenced class
(optional)
.getParentRef(): True if this property references an object in
which this object is in a child-parent
relationship.
{noformat}
h4. Schema for describing a Statistic
The SchemaStatistic class represents the description of a statistic
datum. Once instantiated, the SchemaStatistic is immutable.
{noformat}
class SchemaStatistic:
<constructor>( name=<name-value>,
type=<type-value>,
... )
.getName(): name of the statistic (string)
.getType(): typecode for statistic
.getUnit(): string describing units (optional)
.getDesc(): string description of this statistic (optional)
{noformat}
h4. Schema for describing Method Calls
The Schema for describing a method call involves two classes. The
SchemaMethod class describes the method's structure, and contains a
SchemaArgument class for each argument declared by the method.
An Agent may construct a SchemaMethod's argument list.
{noformat}
class SchemaMethod:
<constructor>( name=<name-value>,
[args=<list of SchemaArguments>]
.getName(): name of method
.getArgCount(): count of arguments
.getArguments(): returns a list of SchemaArgument instances, one
for each argument.
.getArg(index): return the SchemaArgument at index
.addArgument(SchemaArgument): adds a SchemaArgument
class SchemaArgument:
<constructor>( name=<name-value>,
type=<type-value>,
...)
.getName(): name of argument
.getType(): type of argument
.getDirection(): "I"=input, "O"=output, or "IO"=input/output
.getUnit(): string describing units (optional)
.getMin(): minimum value (optional)
.getMax(): maximum value (optional)
.getMaxLen(): maximum length for string values (optional)
.getDesc(): string description of this argument (optional)
.getDefault(): default value.
{noformat}
h4. Schema for Data Objects and Events
A top-level data object in QMF is a collection of statistics,
properties, and methods. Data objects are described by the class
SchemaObjectClass. Data objects may be managed (owned) by an Agent.
Events describe a change within the underlying management system. Unlike
data objects, events do not correspond to a managable entity. They
are composed of a list of arguments that apply to the event. Event
objects are described by the class SchemaEventClass.
Both of these classes derive from the virtual base SchemaClass.
Agent applications may dynamically construct instances of these
objects by adding properties/statistics/methods/etc. at run time.
However, once the Schema is published, it must be considered
immutable, as the hash value must be constant once the Schema is
in use.
{noformat}
class SchemaClass:
.getType(): return SchemaTypeObject or SchemaTypeEvent
.getClassKey(): return the SchemaClassKey that identifies this
Schema instance.
.generateHash(): generate a hash over the body of the schema,
and return a string representation of the hash
in format "%08x-%08x-%08x-%08x"
class SchemaObjectClass(SchemaClass):
<constructor>(?tbd?)
.getPropertyCount(): return the number of properties.
.getStatisticCount(): return the number of statistics.
.getMethodCount(): return the number of methods.
.getProperty(index): return the index'th SchemaProperty.
.getStatistic(index): return the index'th SchemaStatistic.
.getMethod(index): return the index'th SchemaMethod
.addProperty(SchemaProperty): append a new property
.addStatistic(SchemaStatistic): append a new statistic
.addMethod(SchemaMethod): append a new method.
class SchemaEventClass(SchemaClass):
<constructor>(?tbd?)
.getArgCount(): return the count of arguments.
.getArgument(index): return the index'th SchemaArgument.
.addArgument(SchemaArgument): append a new argument.
{noformat}
h3. Data Model
The QMF defines three _layers_ of data representation:
# arbitrarily structured data
# structured data
# managed structured data
Arbitrarily structured data is represented by a map of named data
items. There is no schema associated with this data, as its structure
is arbitrary. This class of data is represented by the QmfData class.
Data that has a formally defined structure is represented by the
QmfDescribed class. This class extends the QmfData class by
associating the data with a formal schema.
Managed structured data extends the concept of structured data by
providing an owner for the data. In QMF, this owner is an Agent.
This class of data extends the QmfDescribed class by adding an
identifier of the owning Agent.
h4. QmfData Class
{noformat}
class QmfData:
<constructor>( map of "name=value" pairs )
{noformat}
h4. QmfDescribed Class
{noformat}
class QmfDescribed(QmfData):
<constructor>( class QmfData, class SchemaClassKey )
.getSchemaClassKey(): returns the identifier of the Schema that
describes the structure of the data.
.getIndex(): return a string composed from "index" property
values.
.getProperties(): return a list of (SchemaProperty, value) pairs
for each of the object's properties.
.getStatistics(): return a list of (SchemaStatistics, value)
pairs for each of the object's statistics.
{noformat}
h4. QmfManaged Class
The QmfManaged class represents a data object that is owned by a
particular Agent. All QmfManaged objects contain a unique object
identifier.
The QmfManaged class extends the QmfDescribed class:
{noformat}
class QmfManaged(QmfDescribed):
.getObjectId(): return ObjectId for this object.
{noformat}
h5. Object Identification
An instance of managed object must be uniquely identified within the
management system. Each managed object is given a name that is unique
within the domain of the object's managing Agent. Note that these
names are not unique across Agents. Therefore, a globally unique name
for an instance of a managed object is the concatenation of the
object's name and the managing Agent's name. This unique object
identifier is represented by the ObjectId class.
ObjectIds are considered _hashable_. That is, it is a requirement
that ObjectIds are deterministically sortable.
{noformat}
class ObjectId:
<constructor>(agent="agent name",
name="object name")
.getAgentName(): returns <agent name> string.
.getObjectName(): returns <object name> string.
.operators(==, !=, <, >) supported.
{noformat}
h4. QmfEvent Class
A QMF Event is a type of QmfDescribed data that is not managed.
Events are notifications that are sent by Agents.
Unlike QmfManaged objects, events do not correspond to managed data on the
Agent. Instead, an event notifies a Console of a change in some
aspect of the system under managment. For example, an event may
indicate that a threshold was exceeded, or a resource was returned.
The structure of an event is described by the SchemaEventClass. An
instance of an event is represented by the QmfEvent class.
{noformat}
class QmfEvent(QmfDescribed):
<constructor>( timestamp, argument_list )
.getTimestamp(): return a timestamp indicating when the Event
triggered.
.getArguments(): return a list of (SchemaArgument, value) pairs
.getArgument(<name>): return the value of argument <name>
?tbd? do we need to identify the originating Agent?
{noformat}
h3. Data Management
The role of a QMF component determines how it will interact with
managment data. Therefore the QmfManaged class is subclassed
to provide an Agent specific implementation and a Console specific
implementation.
The Console application represents a managed data object by the
QmfConsoleData class. The Console has "read only" access to the
statistics and properties in the data object via this class. The
Console can also invoke the methods defined by the object via this
class. The actual data stored in this object is cached from the
Agent. In order to update the cached values, the Console invokes the
refresh() method.
Note that the refresh() and invokeMethod() methods require
communication with the remote Agent. As such, they may block. For
these two methods, the Console has the option of blocking in the call
until the call completes. Optionally, the Console can receive a
notification asynchronously when the operation is complete. See below
for more detail regarding synchronous and asynchronous API calls.
{noformat}
class QmfConsoleData(QmfManaged):
<constructor>(?tbd?)
.getTimestamps(): returns a list of timestamps describing the
lifecycle of the object:
[0] = time of last update from Agent,
[1] = creation timestamp
[2] = deletion timestamp, or zero if not deleted.
.isDeleted(): True if deletion timestamp not zero.
.refresh([reply-handle | timeout]): request that the Agent
update the value of this object's contents.
.invokeMethod(name, inArgs[], [[reply-handle] | [timeout]]):
invoke the named method.
.getStatistic(name): return the value of the named statistic
.getProperty(name): return the value of the named property.
{noformat}
The Agent that manages the data represents it by the
QmfAgentData class. The Agent is responsible for managing the
values of the properties and statistics within the object, as well as
servicing the object's method calls. Unlike the Console, the Agent
has full control of the state of the object.
More ?tbd?
{noformat}
class QmfAgentObject(QmfObject):
<constructor>( ObjectId, ?tbd?)
.destroy(): mark the object as deleted by setting the deletion
timestamp to the current time.
.setObjectId(ObjectId): set the value of the ObjectId.
.setProperty(name, value): update the value of the property.
.setStatistic(name, value): set the value of a statistic.
.incStatistic(name, delta): add the delta to the statistic.
.decStatistic(name, delta): subtract the delta from the statistic.
?tbd?
{noformat}
Note that some languages will allow properties, statistics, and
methods to act as attributes of the QmfObject itself. As an example,
assume an QmfConsoleObject exists with a property named "state". In
python, it will be possible to access the value of "state" as if it
where an attribute of the QmfConsoleObject. The following two
statements are equivalent:
{noformat}
print("My state is '%s'" %
myQmfConsoleObject.getProperty("state"))
print("My state is '%s'" %
myQmfConsoleObject.state)
{noformat}
Be aware - a QmfConsoleObject provides read-only access, so the
following statement would be illegal:
{noformat}
myQmfConsoleObject.state = "up"
{noformat}
h5. Invoking Methods
A managed object's methods provide a mechanisms for a Console application to
perform a "remote procedure call" against the object. The method
actually executes on the Agent, and the result is passed back to the
Console on completion.
The value(s) returned to the Console when the method call completes
are represented by the MethodResult class. The MethodResult class
indicates whether the method call succeeded or not, and, on success,
provides access to all data returned by the method call.
Should a method call result in a failure, this failure is indicated by
the presence of a QmfData object in the MethodResult. This object is
described as an "exception", and contains a description of the reason
for the failure. There are no returned arguments when a method call fails.
A successful method invokation is indicated by the absence of an
exception in the MethodResult. In addition, a map of returned
argument values may be present. The structure of the returned
arguments is described by the SchemaArgument list associated with the
MethodSchema that defines the method. The map contains only those
arguments that the SchemaArgument marks with an "output" direction.
{noformat}
class MethodResult:
<constructor>( QmfData <exception> | <argument-map> )
.getException(): returns exception data if method fails.
.getArgumentCount(): returns number of arguments.
.getArguments(): returns a list of (SchemaArgument, value) pairs
for each returned argument.
.getArgument(<name>): returns value of argument named "name".
{noformat}
h4. Queries
A Query is a mechanism for interrogating the management database. A
Query represents a selector which is sent to an Agent. The Agent
applies the Query against the management database, and
returns those objects which meet the constraints contained in the query.
{noformat}
class Query:
<constructor>(<SchemaClassKey> |
<ObjectId> |
<package-name>[, <class-name>])
.getSchemaClassKey()
.getObjectId()
.getPackageName()
.getClassName()
{noformat}
The target for a Query is determined by the constructor. When a Query
is created using a SchemaClassKey, it will match against the
SchemaClass identified by that key. When a Query is created using an
ObjectId, it will match against the QmfObject or Event identified by
that ObjectId. When a Query is created with only a package name, it
will return all class names that belong to that package. If both the
package and class names are provided, the Query will match against the
SchemaClassKey identified by the combination of those names.
h4. Subscriptions
A subscription is a mechanism for monitoring management data for
changes in its state. A Console creates a subscription with an Agent
based on a Query. The Query specifies the set of management data that
is to be monitored. When the Agent detects changes to the selected
set, a notification is sent to the subscribing Console(s). A
Subscription is represented by the SubscriptionId class. A Console
must cancel the subscription when the console no longer needs to
monitor the data.
{noformat}
class SubscriptionId:
?tbd?
{noformat}
h4. Agent Identifiers
An Agent is uniquely identified by an AgentId. An AgentId is a
three-tuple containing:
* the name of the vendor that produced the agent
* the name of the product using the agent
* the name of the agent component within the product.
This naming convention allows for a single product to host multiple
distinct Agents. The Agent identifier is represented by the AgentId
class. An AgentId is considered _hashable_. That is, it is a
requirement that AgentIds are deterministically sortable.
{noformat}
class AgentId:
<constructor>( vendor, product, name )
.getVendor(): return the vendor name string
.getProduct(): return the product name string
.getName(): return agent component name string
{noformat}
By convention, AgentIds are specified as a string containing the
vendor, product, and name strings separated by colons ":", in that
order. For example: "company.com:OurProduct:AnAgent"
h2. Console application model
A QMF console component is represented by a Console class. This class
is the topmost object of the console application's object model.
A Console is composed of the following objects:
* a connection to the AMQP bus
* a queue of inbound work items
* a collection of all known schemas
* a list of all known remote Agents
* a cache of QmfConsoleObject proxies
The connection to the AMQP bus is used to communicate with remote
Agents. The queue is used as a source for notifications coming from
remote Agents.
h3. Asychronous event model.
The original QMF API defined a set of callback methods that a Console
application needed to provide in order to process asynchronous QMF
events. Each asynchonous event defined its own callback method.
The new API replaces this callback model with a work-queue approach.
All asynchronous events are represented by a WorkItem object. When
a QMF event occurs it is translated into a WorkItem object and placed
in a FIFO queue. It is left to the console application to drain
this queue as needed.
This new API does require the console application to provide a single
callback. The callback is used to notify the console application that
WorkItem object(s) are pending on the work queue. This callback is
invoked by QMF when the work queue transitions from the empty to the
non-empty state. To avoid any potential threading issues, the console
application is _not_ allowed to call any QMF API from within the
callback context. The purpose of the callback is to allow the console
application to schedule itself to drain the work queue at the next
available opportunity.
For example, a console application may be designed using a {{select()}}
loop. The application waits in the {{select()}} for any of a number
of different descriptors to become ready. In this case, the callback
could be written to simply make one of the descriptors ready, and then
return. This would cause the application to exit the wait state, and
start processing pending events.
The callback is represented by the Notifier virtual base class. This
base class contains a single method. A console application derives a
custom handler from this class, and makes it available to the Console
object.
{noformat}
class Notifier:
.consoleIndication(): Called when the console's internal work
queue becomes non-empty due to the arrival of one or more
WorkItems. This method may be called by the internal console
management thread - it is illegal to invoke any QMF APIs from
within this callback. Its purpose is to indicate that the console
application should schedule itself to process the work items.
{noformat}
The WorkItem class represents a single notification event that is read
from the work queue:
{noformat}
class WorkItem:
#
# Enumeration of the types of WorkItems produced by the Console
#
AGENT_ADDED = 1
AGENT_DELETED = 2
NEW_PACKAGE = 3
NEW_CLASS = 4
OBJECT_UPDATE = 5
EVENT_RECEIVED = 7
AGENT_HEARTBEAT = 8
.getType(): Identifies the type of work item by returning one of
the above type codes.
.getHandle(): return the handle for an asynchronous operation, if present.
.getParams(): Returns the data payload of the work item. The type
of this object is determined by the type of the workitem (?TBD?).
{noformat}
h3. Local representation of a remote Agent.
The console application maintains a list of all known remote Agents.
Each Agent is represented by the Agent class:
{noformat}
class Agent:
<constructor>( AgentId )
.getName(): returns the AgentId
?tbd?
{noformat}
h3. The Console Object.
The Console class is the top-level object used by a console
application. All QMF console functionality is made available by this
object. A console application must instatiate one of these objects.
As noted below, some Console methods require contacting a remote
Agent. For these methods, the caller has the option to either block
for a (non-infinite) timeout waiting for a reply, or to allow the
method to complete asynchonously. When the asynchronous approach is
used, the caller must provide a unique handle that identifies the
request. When the method eventually completes, a WorkItem will be
placed on the work queue. The WorkItem will contain the handle that
was provided to the corresponding method call.
All blocking calls are considered thread safe - it is possible to have
a multi-threaded implementation have multiple blocking calls in flight
simultaineously.
If a name is supplied, it must be unique across all Consoles attached
to the AMQP bus. If no name is supplied, a unique name will be
synthesized in the format: "qmfc-<hostname>.<pid>"
{noformat}
class Console:
<constructor>(name=<name-str>,
notifier=<class Notifier>,
timeout=<default for all blocking calls>,
subscription_duration=<default lifetime of a subscription>)
.destroy(timeout=None): Must be called to release Console's resources.
.addConnection(QPID Connection): Connect the console to the AMQP cloud.
.removeConnection(conn): Remove the AMQP connection from the
console. Un-does the addConnection() operation, and
releases any agents associated with the connection. All
blocking methods are unblocked and given a failure status.
All outstanding asynchronous operations are cancelled
without producing WorkItems.
.getAddress():
Get the AMQP address this Console is listening to (type str).
.findAgent( class AgentId, [timeout | handle] ): Query for the
presence of a specific agent. Returns a class Agent if the agent
is present. May be called blocking (with default timeout
override), or may allow asynchronous completion (producing a
WorkItem with the given handle).
.enableAgentDiscovery(): Called to enable the asynchronous
Agent Discovery process. Once enabled, AGENT_ADD work items
can arrive on the WorkQueue.
.disableAgentDiscovery(): Called to disable the async Agent
Discovery process enabled by calling enableAgentDiscovery().
.getWorkItemCount(): Returns the count of pending WorkItems that
can be retrieved.
.getNextWorkItem([timeout=0]): Obtains the next pending work
item, or None if none available.
.releaseWorkItem(wi): Releases a WorkItem instance obtained by
getNextWorkItem(). Called when the application has finished
processing the WorkItem.
.getAgents(): Returns a list of available agents (class Agent)
.getAgent( class AgentId ): Return the class Agent for the
named agent, if known.
.getPackages( [class Agent] ): Returns a list of the names of
all known packages. If an optional Agent is provided, then
only those packages available from that Agent are returned.
.getClasses( [class Agent] ): Returns a list of SchemaClassKeys
for all available Schema. If an optional Agent is provided,
then the returned SchemaClassKeys are limited to those
Schema known to the given Agent.
.getSchema( class SchemaClassKey [, class Agent]): Return a list
of all available class SchemaClass across all known agents.
If an optional Agent is provided, restrict the returned
schema to those supported by that Agent.
.makeObject( SchemaClassKey, **kwargs ): returns an uninitialized
instance of a QmfDescribed data object.
.getObjects( _SchemaClassKey= | _package=, _class= |
_objectId=,
[timeout=],
[list-of-class-Agent] ): perform a blocking query
for QmfConsoleObjects. Returns a list (possibly empty) of matching
objects. The selector for the query may be either:
* class SchemaClassKey - all objects whose schema match the
schema identified by _SchemaClassKey parameter.
* package/class name - all objects whose schema are
contained by the named package and class.
* the object identified by _objectId
This method will block until all known agents reply, or the
timeout expires. Once the timeout expires, all
data retrieved to date is returned.
If a list of agents is supplied, then the query is sent to
only those agents.
.createSubscription( class Query [, duration=<secs> [, list of agents] ): creates a
subscription using the given Query. If a list of agents
is provided, the Query will apply only to those agents.
Otherwise it will apply to all active agents, including
those discovered during the lifetime of the subscription.
The duration argument can be used to override the
console's default subscription lifetime for this
subscription. Returns a class SubscriptionId.
.refreshSubscription( SubscriptionId [, duration=<secs>] ):
(re)activates a subscription. Uses the console default duration
unless the duration is explicitly specified.
.cancelSubscription( SubscriptionId ): terminates the
subscription.
{noformat}
h2. Example Console Application
The following pseudo-code performs a blocking query for a particular agent.
{noformat}
logging.info( "Starting Connection" )
conn = Connection("localhost")
conn.connect()
logging.info( "Starting Console" )
myConsole = Console()
myConsole.addConnection( conn )
logging.info( "Finding Agent" )
myAgent = myConsole.findAgent( AgentId( "aCompany.com", "Examples", "anAgent" ), _timeout=5 )
if myAgent:
logging.info( "Agent Found: %s" % myAgent )
else:
logging.info( "No Agent Found!")
logging.info( "Removing connection" )
myConsole.removeConnection( conn )
logging.info( "Destroying console:" )
myConsole.destroy( _timeout=10 )
{noformat}
The following pseudo-code performs a non-blocking query for all
agents. It completes when at least one agent is found.
{noformat}
class MyNotifier(Notifier):
def __init__(self, context):
self._myContext = context
self.WorkAvailable = False
def console_indication(self):
print("Indication received! context=%d" % self._myContext)
self.WorkAvailable = True
noteMe = MyNotifier( 668 )
logging.info( "Starting Connection" )
conn = Connection("localhost")
conn.connect()
myConsole = Console(notifier=noteMe)
myConsole.addConnection( conn )
myConsole.enableAgentDiscovery()
logging.info("Waiting...")
while not noteMe.WorkAvailable:
print("No work yet...sleeping!")
time.sleep(1)
print("Work available = %d items!" % myConsole.getWorkItemCount())
wi = myConsole.getNextWorkitem(timeout=0)
while wi:
print("work item %d:%s" % (wi.getType(), str(wi.getParams())))
wi = myConsole.getNextWorkitem(timeout=0)
logging.info( "Removing connection" )
myConsole.remove_connection( conn )
logging.info( "Destroying console:" )
myConsole.destroy( 10 )
{noformat}
h2. Revision History
10/20/2009 - First Revision
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [QMF] Proposed API for QMFv2 - RFC
Posted by Ted Ross <tr...@redhat.com>.
Thanks Ken,
I've posted your wiki text at
http://cwiki.apache.org/confluence/display/qpid/QMFv2+API+Proposal.
-Ted
On 11/20/2009 01:47 PM, Ken Giusti wrote:
> Hi All,
>
> Below is a strawman proposal for the design of the QMFv2 API. It describes an api for working with schema, the data model, and console-side operations. There's a few tbd's in it, and I still need to think about the Agent side, but I'd like to get some feedback on it early.
>
> Would it be possible to post this as a rough draft on the QMFv2 API section of the wiki? I've tried to use confluence markup - but since I cannot submit I have no idea how (badly) it renders.
>
> Thanks!
>
> -K
>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org