You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by lu...@redhat.com on 2011/11/03 00:08:13 UTC
Sketch of CIMI model layer
This patch is a first attempt at a CIMI model layer. Such a layer is needed
to deal with deserialization (and possibly serialization) of CIMI model
objects from (to) XML/JSON, and to give us a place to stash business logic
connected to model classes.
This patch is solely for discussion, it most definitely will not work; at a
minimum there are various 'require' that are missing.
There's two things that make such a model layer interesting:
* The need to deal with differences between the JSON and XML
representation; I believe they are all mechanical, though the CIMI
draft does not set out any explicit rules. The main rule is that an
array of things in JSON is represented as { 'things': [ .. array .. ] }
whereas the XML uses a sequence of <thing/> elements
* The desire to keep the metadata that has to be written for each model
class minimal
Since we need to capture some of the metadata in the model classes anyway
(at a bare minimum we need to know which attributes are arrays, so that we
can handle JSON and XML differently), I think it will also be possible to
do serializiation of model classes programmatically, rather than through
templates.
The intended use of the model classes is something like (of course, in
reality, this code will be scattered through various places like server.rb
and the drivers)
templ = nil
if input_is_xml(input)
templ = MachineTemplate.from_xml(input)
else
templ = MachineTemplate.from_json(input)
end
machine = Machine.new
machine.name = templ.name
machine.volumes = templ.volumes.dup
templ.volume_templates.each do |vt|
machine.volumes << setup_volume_from_template(vt)
end
# Do more stuff with templ, adding to machine
respond_to do |fmt|
fmt.xml { machine.to_xml }
fmt.json { machine.to_json }
end
Besides getting the code to actually be syntactically correct, there are a
few more additions that seem useful:
* make sure we strip out unneeded arrays from XmlSimple (i.e., some
scalars are returned as '[ v ]', but we should only use 'v'
* add a way to the DSL to indicate whether an attribute is mandatory or
optional, i.e. something like
ref :uri, :mandatory => true
With that, we could do some rudimentary validation on model objects
* see if we can really have a Base.to_xml; the two big issues to settle
are
(1) indicate which attributes go into XML attributes and which ones
go into elements, e.g.
ref :foo, :xml_attr => true
(2) for attributes that are not scalars, do (1) recursively; in
particular, allow associating a class with the members of an
array
array :things, :class => Thing
where Thing itself is a subclass of Base
* some type specific validation (a 'ref' should look like a URI, we could
add types to 'scalar' and make sure integers look like integers etc.)
* a unit test or two
Another sidenote: since the CIMI API isn't terribly HTML friendly (e.g.,
POST's all have XML or JSON bodies), I wonder if the HTML frontend
shouldn't be a JS app that communicates with the server via JSON only.
David
Re: Sketch of CIMI model layer
Posted by Francesco Vollero <ra...@gmail.com>.
Sent from my iPad
On 03/nov/2011, at 00:08, lutter@redhat.com wrote:
>
> This patch is a first attempt at a CIMI model layer. Such a layer is needed
> to deal with deserialization (and possibly serialization) of CIMI model
> objects from (to) XML/JSON, and to give us a place to stash business logic
> connected to model classes.
>
>
>
> Another sidenote: since the CIMI API isn't terribly HTML friendly (e.g.,
> POST's all have XML or JSON bodies), I wonder if the HTML frontend
> shouldn't be a JS app that communicates with the server via JSON only.
Indeed, is more easy to manage JSON Objects from Javascript, and is quite simple as well.
cheers,
Francesco
>
> David
Re: Sketch of CIMI model layer
Posted by David Lutterkort <lu...@redhat.com>.
Hi Michal,
thanks for the patches. I folded all but the first one into mine.
On Fri, 2011-11-04 at 14:15 +0100, Michal Fojtik wrote:
> I made couple fixes (mostly because it's my way how to get familiar with
> the code ;-). I attached 4 patches:
>
> 0001-CIMI-Fixed-requires-and-added-module.patch
I forgot to git add lib/cimi/model.rb; fixed it by doing that.
> 0003-CIMI-Added-symbolize_key-to-convert-all-string-Hash-.patch
>
> * Usually I prefer Symbol as Hash key instead of String (XmlSimple sux
> there, this patch should add appropriate fix to Hash core_ext)
Yes, agreed; not sure if the extension to Hash is the right way to go,
since we also want to do the camelcase -> underscore conversion for
nested Hashes. I think I'll add a seperate 'object' word to the DSL that
will trigger this conversion, e.g.
> 0004-CIMI-Use-instance-variables-instead-of-getter-setter.patch
>
> * It's better in some (rare) cases to use instance variable when reading
> or storing something. You never know if someone smart enough will not
> redefine the attribute_values getter :)
If they do that, they should suffer ;) But yeah, works for me.
David
Re: Sketch of CIMI model layer
Posted by Michal Fojtik <mf...@redhat.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
David Lutterkort wrote:
> On Thu, 2011-11-03 at 16:20 -0700, David Lutterkort wrote:
>> On Wed, 2011-11-02 at 16:08 -0700, lutter@redhat.com wrote:
>>> This patch is a first attempt at a CIMI model layer. Such a layer
>>> is needed to deal with deserialization (and possibly
>>> serialization) of CIMI model objects from (to) XML/JSON, and to
>>> give us a place to stash business logic connected to model
>>> classes.
>> For those following at home, I uploaded an updated patch to
>> http://people.apache.org/~lutter/patches/cimi-model/ (and I'll
>> continue updating the patch there)
>
> And if you're really following along at home, you can run the
> minimal specs that are there with
>
> cd server rspec spec/cimi/model/machine_template_spec.rb
>
> If you make changes, keep them as a separate patch and send them to
> me, I'll incorporate them.
I made couple fixes (mostly because it's my way how to get familiar with
the code ;-). I attached 4 patches:
0001-CIMI-Fixed-requires-and-added-module.patch
* This was required for executing the rspec command above :-)
0002-CIMI-Fixed-typo-in-base-model.patch
* Seems like wrong assertion to me.
0003-CIMI-Added-symbolize_key-to-convert-all-string-Hash-.patch
* Usually I prefer Symbol as Hash key instead of String (XmlSimple sux
there, this patch should add appropriate fix to Hash core_ext)
0004-CIMI-Use-instance-variables-instead-of-getter-setter.patch
* It's better in some (rare) cases to use instance variable when reading
or storing something. You never know if someone smart enough will not
redefine the attribute_values getter :)
I drop a quick note in the last one. I suggest to use HAML views to
produce output XML instead of returning XML programmatically. In worst
case we can use something like:
engine = Tilt::HamlTemplate.new(File.join(self.name, 'index.xml.haml'))
engine.render(self, :'@entity' => self)
to produce XML output.
-- Michal
- --
Michal Fojtik, mfojtik@redhat.com
Deltacloud API: http://deltacloud.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJOs+VpAAoJEEnFpmY+LvFRnMQH/0Goc7Y/6f1tCSiyOrqFtIuB
XrTB6OSPGSSfcZcEskY+iQR6gv/Jth56HpEZkqHNPIqd/0o0y/ZKHpRxSnVGnMIh
MV+9CLMxwsDGfYjUGm6NrHjhzMMvqcLwHrxP38+u8Adu0KkOIEixS5jh06n9tZve
G/5v6h6EvrHK1XhwFa1BJD/cBzcyZh54ICsb32UGsjwwbnlerhk1xkzm4NduB70K
DhWinF1jDpbwc8+0gh4+olumwoVePtsxQZZe3Jpj0Cu/W3Fz6RuzqfM55F42mOyL
TaMLUF5fokVDK9rZ/3Cry8O1RIbLCYRgNF/hdSTiPo3OKiAl3nXMyuGnplAMyF4=
=kVZK
-----END PGP SIGNATURE-----
Re: Sketch of CIMI model layer
Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2011-11-03 at 16:20 -0700, David Lutterkort wrote:
> On Wed, 2011-11-02 at 16:08 -0700, lutter@redhat.com wrote:
> > This patch is a first attempt at a CIMI model layer. Such a layer is needed
> > to deal with deserialization (and possibly serialization) of CIMI model
> > objects from (to) XML/JSON, and to give us a place to stash business logic
> > connected to model classes.
>
> For those following at home, I uploaded an updated patch to
> http://people.apache.org/~lutter/patches/cimi-model/ (and I'll continue
> updating the patch there)
And if you're really following along at home, you can run the minimal
specs that are there with
cd server
rspec spec/cimi/model/machine_template_spec.rb
If you make changes, keep them as a separate patch and send them to me,
I'll incorporate them.
David
Re: Sketch of CIMI model layer
Posted by David Lutterkort <lu...@redhat.com>.
On Wed, 2011-11-02 at 16:08 -0700, lutter@redhat.com wrote:
> This patch is a first attempt at a CIMI model layer. Such a layer is needed
> to deal with deserialization (and possibly serialization) of CIMI model
> objects from (to) XML/JSON, and to give us a place to stash business logic
> connected to model classes.
For those following at home, I uploaded an updated patch to
http://people.apache.org/~lutter/patches/cimi-model/ (and I'll continue
updating the patch there)
David
Re: Sketch of CIMI model layer
Posted by Doug Davis <du...@us.ibm.com>.
Perhaps we should discuss this in the CMWG. I'm not sure we've ever
really had an in-depth discussion on this topic with the entire team.
To me, given that JSON doesn't have an outermost wrapper telling people
what kind of entity is being serialized I think having the classname
someplace in the mime type is important. Whether its:
application/CIMI-Machine+json or application/json;type=CIMI-Machine
doesn't really matter much to me.
thanks
-Doug
______________________________________________________
STSM | Standards Architect | IBM Software Group
(919) 254-6905 | IBM 444-6905 | dug@us.ibm.com
The more I'm around some people, the more I like my dog.
David Lutterkort <lu...@redhat.com>
11/03/2011 01:04 PM
To
deltacloud-dev@incubator.apache.org
cc
Doug Davis/Raleigh/IBM@IBMUS
Subject
Re: Sketch of CIMI model layer
On Thu, 2011-11-03 at 08:56 -0400, Tong Li wrote:
> In my previous implementation, request with body are always in xml
format,
> it was never in a HTML form post. The reason was that DMTF spec did not
> really care HTML. All forms of requests should be in theory the
> content-type of application/CIMI-<ResourceType>+xml or
> application/CIMI-<ResourceType>+json. When I implemented the client to
> produce the request, I did not add that header, because I was not sure
if
> add that header will actually break Sinatra so it may not be able to
> determine if the request is actually for xml or json.
That's actually a very good point: these custom MIME types are a pain in
the neck. They break the default behavior of any
dispatch-by-content-type framework, and require some manual fiddling to
get it back on track.
In Deltacloud, we generate the response with the following (in Rails,
the code would look exactly the same):
respond_to do |format|
format.xml { .. generate XML output .. }
format.json { .. generate JSON output .. }
end
With the custom MIME types in the Accept header, the above doesn't work
out of the box - we painstakingly have to teach the framework that
application/CIMI-Foo+xml is really application/xml, and
application/CIMI-Foo+json is application/json.
The purported gain of being able to dispatch on/know what is being asked
for is not there. To me the whole thing is a little bit like replacing
image/png with image/passport-pic+png, image/soccer-team+png,
image/kittens-in-a-pile+png, etc.
If there ever is a need for the client to select the representation of a
resource by type, this can be done through media type parameters, i.e.
the client could send
Accept:
application/xml;type=BriefMachine,application/xml;q=0.8;type=FullMachine
But I don't really see a need for that - there is only ever one CIMI
Type that comes back from a request, and I don't believe that the
flexibility of choosing between types will really help implementors.
David
Re: Sketch of CIMI model layer
Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2011-11-03 at 08:56 -0400, Tong Li wrote:
> In my previous implementation, request with body are always in xml format,
> it was never in a HTML form post. The reason was that DMTF spec did not
> really care HTML. All forms of requests should be in theory the
> content-type of application/CIMI-<ResourceType>+xml or
> application/CIMI-<ResourceType>+json. When I implemented the client to
> produce the request, I did not add that header, because I was not sure if
> add that header will actually break Sinatra so it may not be able to
> determine if the request is actually for xml or json.
That's actually a very good point: these custom MIME types are a pain in
the neck. They break the default behavior of any
dispatch-by-content-type framework, and require some manual fiddling to
get it back on track.
In Deltacloud, we generate the response with the following (in Rails,
the code would look exactly the same):
respond_to do |format|
format.xml { .. generate XML output .. }
format.json { .. generate JSON output .. }
end
With the custom MIME types in the Accept header, the above doesn't work
out of the box - we painstakingly have to teach the framework that
application/CIMI-Foo+xml is really application/xml, and
application/CIMI-Foo+json is application/json.
The purported gain of being able to dispatch on/know what is being asked
for is not there. To me the whole thing is a little bit like replacing
image/png with image/passport-pic+png, image/soccer-team+png,
image/kittens-in-a-pile+png, etc.
If there ever is a need for the client to select the representation of a
resource by type, this can be done through media type parameters, i.e.
the client could send
Accept: application/xml;type=BriefMachine,application/xml;q=0.8;type=FullMachine
But I don't really see a need for that - there is only ever one CIMI
Type that comes back from a request, and I don't believe that the
flexibility of choosing between types will really help implementors.
David
Re: Sketch of CIMI model layer
Posted by Tong Li <li...@us.ibm.com>.
Tong Li
Emerging Technologies & Standards
Building 501/B205
litong01@us.ibm.com
lutter@redhat.com wrote on 11/02/2011 07:08:13 PM:
> From: lutter@redhat.com
> To: deltacloud-dev@incubator.apache.org
> Cc: Doug Davis/Raleigh/IBM@IBMUS
> Date: 11/02/2011 07:09 PM
> Subject: Sketch of CIMI model layer
>
>
> This patch is a first attempt at a CIMI model layer. Such a layer is
needed
> to deal with deserialization (and possibly serialization) of CIMI model
> objects from (to) XML/JSON, and to give us a place to stash business
logic
> connected to model classes.
>
> This patch is solely for discussion, it most definitely will not work; at
a
> minimum there are various 'require' that are missing.
>
> There's two things that make such a model layer interesting:
>
> * The need to deal with differences between the JSON and XML
> representation; I believe they are all mechanical, though the CIMI
> draft does not set out any explicit rules. The main rule is that an
> array of things in JSON is represented as { 'things': [ ..
array .. ] }
> whereas the XML uses a sequence of <thing/> elements
> * The desire to keep the metadata that has to be written for each model
> class minimal
>
> Since we need to capture some of the metadata in the model classes anyway
> (at a bare minimum we need to know which attributes are arrays, so that
we
> can handle JSON and XML differently), I think it will also be possible to
> do serializiation of model classes programmatically, rather than through
> templates.
>
> The intended use of the model classes is something like (of course, in
> reality, this code will be scattered through various places like
server.rb
> and the drivers)
>
> templ = nil
> if input_is_xml(input)
> templ = MachineTemplate.from_xml(input)
> else
> templ = MachineTemplate.from_json(input)
> end
> machine = Machine.new
>
> machine.name = templ.name
> machine.volumes = templ.volumes.dup
> templ.volume_templates.each do |vt|
> machine.volumes << setup_volume_from_template(vt)
> end
>
> # Do more stuff with templ, adding to machine
> respond_to do |fmt|
> fmt.xml { machine.to_xml }
> fmt.json { machine.to_json }
> end
>
> Besides getting the code to actually be syntactically correct, there are
a
> few more additions that seem useful:
>
> * make sure we strip out unneeded arrays from XmlSimple (i.e., some
> scalars are returned as '[ v ]', but we should only use 'v'
> * add a way to the DSL to indicate whether an attribute is mandatory or
> optional, i.e. something like
>
> ref :uri, :mandatory => true
>
> With that, we could do some rudimentary validation on model objects
>
> * see if we can really have a Base.to_xml; the two big issues to settle
> are
> (1) indicate which attributes go into XML attributes and which ones
> go into elements, e.g.
> ref :foo, :xml_attr => true
>
> (2) for attributes that are not scalars, do (1) recursively; in
> particular, allow associating a class with the members of an
> array
>
> array :things, :class => Thing
>
> where Thing itself is a subclass of Base
>
> * some type specific validation (a 'ref' should look like a URI, we
could
> add types to 'scalar' and make sure integers look like integers etc.)
>
> * a unit test or two
>
> Another sidenote: since the CIMI API isn't terribly HTML friendly (e.g.,
> POST's all have XML or JSON bodies), I wonder if the HTML frontend
> shouldn't be a JS app that communicates with the server via JSON only.
In my previous implementation, request with body are always in xml format,
it was never in a HTML form post. The reason was that DMTF spec did not
really care HTML. All forms of requests should be in theory the
content-type of application/CIMI-<ResourceType>+xml or
application/CIMI-<ResourceType>+json. When I implemented the client to
produce the request, I did not add that header, because I was not sure if
add that header will actually break Sinatra so it may not be able to
determine if the request is actually for xml or json.
>
> David
>
[PATCH] Implementation sketch: model layer for CIMI
Posted by lu...@redhat.com.
From: David Lutterkort <lu...@redhat.com>
Signed-off-by: David Lutterkort <lu...@redhat.com>
---
server/lib/cimi/model/base.rb | 145 +++++++++++++++++++++++++++++
server/lib/cimi/model/machine_template.rb | 21 ++++
2 files changed, 166 insertions(+), 0 deletions(-)
create mode 100644 server/lib/cimi/model/base.rb
create mode 100644 server/lib/cimi/model/machine_template.rb
diff --git a/server/lib/cimi/model/base.rb b/server/lib/cimi/model/base.rb
new file mode 100644
index 0000000..5bc0677
--- /dev/null
+++ b/server/lib/cimi/model/base.rb
@@ -0,0 +1,145 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership. The
+# ASF licenses this file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance with the
+# License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+class Base
+
+ #
+ # Attributes describe how we extract values from XML/JSON
+ #
+ class Attribute
+ attr_reader :name
+
+ def initialize(name)
+ @name = name
+ end
+
+ def from_xml(values, xml)
+ values[name] = xml[name]
+ end
+
+ def from_json(values, json)
+ values[name] = json[name]
+ end
+ end
+
+ class Scalar < Attribute; end
+ class Ref < Scalar; end
+ class Array < Attribute
+ # For an array :things, we collect all <thing/> elements (XmlSimple
+ # actually does the collecting)
+ def from_xml(values, xml)
+ values[@name] = xml[@name.to_s.singularize.to_sym]
+ end
+ end
+
+ #
+ # We keep the values of the attributes in a hash
+ #
+ attr_reader :attribute_values
+
+ # Keep the list of all attributes in an array +attributes+; for each
+ # attribute, we also define a getter and a setter to access/change the
+ # value for that attribute
+ class << self
+ def attributes
+ @attributes ||= []
+ end
+
+ def attributes=(a)
+ @attributes = a
+ end
+
+ def inherited(child)
+ child.attributes = self.attributes.dup
+ end
+
+ def add_attributes(names, attr_klass)
+ names.each do |name|
+ attr = attr_klass.new(name)
+ @attributes << attr
+ # Define getter and setter
+ define_method(name) { attribute_values[name] }
+ define_method(:"#{name}=", newval) { attribute_values[name = newval] }
+ end
+ end
+ end
+
+ #
+ # The DSL for declaring various attributes
+ #
+ class << self
+
+ # A reference, i.e., a URI
+ def ref(*args)
+ add_attributes(args, Ref)
+ end
+
+ # A scalar attribute (string, integer, boolean)
+ def scalar(*args)
+ add_attributes(args, Scalar)
+ end
+
+ # An array of values; the type of the values depends on the array, but
+ # is in general a hash
+ def array(*args)
+ add_attributes(args, Array)
+ end
+ end
+
+ #
+ # Factory methods
+ #
+ def initialize(values)
+ @attribute_values = values
+ end
+
+ # Construct a new object from the XML representation +xml+
+ def self.from_xml(xml)
+ h = XmlSimple.xml_in(xml)
+ values = @attributes.inject({}) do |v, attr|
+ attr.from_xml(v, h)
+ v
+ end
+ new(values)
+ end
+
+ # Construct a new object
+ def self.from_json(json)
+ values = @attributes.inject({}) do |v, attr|
+ attr.from_json(v, json)
+ end
+ new(values)
+ end
+
+ #
+ # Serialize
+ #
+ def self.to_json
+ # FIXME: generate hash that is the JSON representation of this object
+ end
+
+ # FIXME: can we do to_xml ? We do not have metadata to distinguish
+ # between attributes and nested elements, especially for array attributes
+ # (e.g., look at <volume/> and <networkInterface/> in MachineTemplate
+
+ #
+ # Common attributes for all resources
+ #
+ ref :uri
+ ref :type_uri
+ scalar :name
+ scalar :created
+ array :properties
+end
diff --git a/server/lib/cimi/model/machine_template.rb b/server/lib/cimi/model/machine_template.rb
new file mode 100644
index 0000000..4b58a87
--- /dev/null
+++ b/server/lib/cimi/model/machine_template.rb
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership. The
+# ASF licenses this file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance with the
+# License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+class MachineTemplate < Deltacloud::CIMI::Model::Base
+ ref :machine_config, :machine_image, :machine_admin
+ array :volumes
+ array :volume_templates
+ array :network_interfaces
+end
--
1.7.6.4
Re: Sketch of CIMI model layer
Posted by "marios@redhat.com" <ma...@redhat.com>.
On 03/11/11 14:44, Tong Li wrote:
> Marios,
> In the spec,for xml, it is normally a list of singulars like
> <volume>...</volume><volume></volume>. For json, it will be "volumes":
> [something, somethingelse].
> I think that was what we discussed. the spec does not have the notion like
> <volumes><volume/><volume/></volumes>
>
yes that's right. I meant, if it will help us to bridge the difference
between the way json and xml are serialised for 'array' attributes (like
<volume>(s) ) then we can try and re-introduce wrappers for those
elements in the xml (i.e. raise an issue to change the spec itself),
marios
> Thanks.
>
> Tong Li
> Emerging Technologies & Standards
> Building 501/B205
> litong01@us.ibm.com
>
> "marios@redhat.com" <ma...@redhat.com> wrote on 11/03/2011 05:46:53 AM:
>
>> From: "marios@redhat.com" <ma...@redhat.com>
>> To: deltacloud-dev@incubator.apache.org
>> Cc: lutter@redhat.com, Doug Davis/Raleigh/IBM@IBMUS
>> Date: 11/03/2011 05:47 AM
>> Subject: Re: Sketch of CIMI model layer
>>
>> On 03/11/11 01:08, lutter@redhat.com wrote:
>>> This patch is a first attempt at a CIMI model layer. Such a layer is
> needed
>>> to deal with deserialization (and possibly serialization) of CIMI model
>>> objects from (to) XML/JSON, and to give us a place to stash business
> logic
>>> connected to model classes.
>>>
>>> This patch is solely for discussion, it most definitely will not work;
> at a
>>> minimum there are various 'require' that are missing.
>>>
>>> There's two things that make such a model layer interesting:
>>>
>>> * The need to deal with differences between the JSON and XML
>>> representation; I believe they are all mechanical, though the CIMI
>>> draft does not set out any explicit rules. The main rule is that an
>>> array of things in JSON is represented as { 'things': [ ..
> array .. ] }
>>> whereas the XML uses a sequence of <thing/> elements
>>
>> I recall discussion a while back about whether we need to include
>> wrappers for 'array' elements in the xml serialization... i.e.
>>
>> <volumes>
>> <volume href="xs:anyURI"
>> attachmentPoint="xs:string" protocol="xs:string" />
>> <volume href="xs:anyURI"
>> attachmentPoint="xs:string" protocol="xs:string" />
>> <volumes/>
>>
>> (not sure but may have been in relation to mantis #1195). I wonder
>> whether this needs to be revisited if it will help to bridge the
>> difference between json+xml 'arrays' and help with our deserialization.
>> The spec doees currently include 'cardinality characters' after items to
>> indicate:
>>
>> "?" (0 or 1)
>> "*" (0 or more)
>> "+" (1 or more)
>>
>> e.g. the xml serialization of the volume attribute for machine template:
>>
>> <volume href="xs:anyURI"
>> attachmentPoint="xs:string" protocol="xs:string" /> *
>>
>> but I don't think that's enough metadata to help us; it does tell us
>> that there may be more of this <thing> but we still need to search
>> through the xml doc to find all occurrences.
>>
>>
>>> * see if we can really have a Base.to_xml; the two big issues to
> settle
>>> are
>>> (1) indicate which attributes go into XML attributes and which
> ones
>>> go into elements, e.g.
>>> ref :foo, :xml_attr => true
>>
>> we could in theory maintain the list of 'entities' though we would need
>> to maintain and update it with the spec... - machine, machine template,
>> volume, network, job, meter etc ... any <thing> that's not an entity is
>> by default an xml attribute. Having said that, the spec does include
>> extensibility mechanisms whereby cloud providers can extend the model by
>> defining new resource types, though these are advertised with
>> 'EntityMetadata' resource (but this last point is a recommendation only).
>>
>>>
>>> (2) for attributes that are not scalars, do (1) recursively; in
>>> particular, allow associating a class with the members of an
>>> array
>>>
>>> array :things, :class => Thing
>>>
>>> where Thing itself is a subclass of Base
>>>
>>> * some type specific validation (a 'ref' should look like a URI, we
> could
>>> add types to 'scalar' and make sure integers look like integers
> etc.)
>>>
>>> * a unit test or two
>>>
>>> Another sidenote: since the CIMI API isn't terribly HTML friendly
> (e.g.,
>>> POST's all have XML or JSON bodies), I wonder if the HTML frontend
>>> shouldn't be a JS app that communicates with the server via JSON only.
>>>
>>> David
>>
Re: Sketch of CIMI model layer
Posted by Tong Li <li...@us.ibm.com>.
Marios,
In the spec,for xml, it is normally a list of singulars like
<volume>...</volume><volume></volume>. For json, it will be "volumes":
[something, somethingelse].
I think that was what we discussed. the spec does not have the notion like
<volumes><volume/><volume/></volumes>
Thanks.
Tong Li
Emerging Technologies & Standards
Building 501/B205
litong01@us.ibm.com
"marios@redhat.com" <ma...@redhat.com> wrote on 11/03/2011 05:46:53 AM:
> From: "marios@redhat.com" <ma...@redhat.com>
> To: deltacloud-dev@incubator.apache.org
> Cc: lutter@redhat.com, Doug Davis/Raleigh/IBM@IBMUS
> Date: 11/03/2011 05:47 AM
> Subject: Re: Sketch of CIMI model layer
>
> On 03/11/11 01:08, lutter@redhat.com wrote:
> > This patch is a first attempt at a CIMI model layer. Such a layer is
needed
> > to deal with deserialization (and possibly serialization) of CIMI model
> > objects from (to) XML/JSON, and to give us a place to stash business
logic
> > connected to model classes.
> >
> > This patch is solely for discussion, it most definitely will not work;
at a
> > minimum there are various 'require' that are missing.
> >
> > There's two things that make such a model layer interesting:
> >
> > * The need to deal with differences between the JSON and XML
> > representation; I believe they are all mechanical, though the CIMI
> > draft does not set out any explicit rules. The main rule is that an
> > array of things in JSON is represented as { 'things': [ ..
array .. ] }
> > whereas the XML uses a sequence of <thing/> elements
>
> I recall discussion a while back about whether we need to include
> wrappers for 'array' elements in the xml serialization... i.e.
>
> <volumes>
> <volume href="xs:anyURI"
> attachmentPoint="xs:string" protocol="xs:string" />
> <volume href="xs:anyURI"
> attachmentPoint="xs:string" protocol="xs:string" />
> <volumes/>
>
> (not sure but may have been in relation to mantis #1195). I wonder
> whether this needs to be revisited if it will help to bridge the
> difference between json+xml 'arrays' and help with our deserialization.
> The spec doees currently include 'cardinality characters' after items to
> indicate:
>
> "?" (0 or 1)
> "*" (0 or more)
> "+" (1 or more)
>
> e.g. the xml serialization of the volume attribute for machine template:
>
> <volume href="xs:anyURI"
> attachmentPoint="xs:string" protocol="xs:string" /> *
>
> but I don't think that's enough metadata to help us; it does tell us
> that there may be more of this <thing> but we still need to search
> through the xml doc to find all occurrences.
>
>
> > * see if we can really have a Base.to_xml; the two big issues to
settle
> > are
> > (1) indicate which attributes go into XML attributes and which
ones
> > go into elements, e.g.
> > ref :foo, :xml_attr => true
>
> we could in theory maintain the list of 'entities' though we would need
> to maintain and update it with the spec... - machine, machine template,
> volume, network, job, meter etc ... any <thing> that's not an entity is
> by default an xml attribute. Having said that, the spec does include
> extensibility mechanisms whereby cloud providers can extend the model by
> defining new resource types, though these are advertised with
> 'EntityMetadata' resource (but this last point is a recommendation only).
>
> >
> > (2) for attributes that are not scalars, do (1) recursively; in
> > particular, allow associating a class with the members of an
> > array
> >
> > array :things, :class => Thing
> >
> > where Thing itself is a subclass of Base
> >
> > * some type specific validation (a 'ref' should look like a URI, we
could
> > add types to 'scalar' and make sure integers look like integers
etc.)
> >
> > * a unit test or two
> >
> > Another sidenote: since the CIMI API isn't terribly HTML friendly
(e.g.,
> > POST's all have XML or JSON bodies), I wonder if the HTML frontend
> > shouldn't be a JS app that communicates with the server via JSON only.
> >
> > David
>
Re: Sketch of CIMI model layer
Posted by Doug Davis <du...@us.ibm.com>.
Ah - that would be a bug :-)
The example in 4.1.10 was just out of sync with the Machine JSON. Will
fix. thanks
thanks
-Doug
______________________________________________________
STSM | Standards Architect | IBM Software Group
(919) 254-6905 | IBM 444-6905 | dug@us.ibm.com
The more I'm around some people, the more I like my dog.
David Lutterkort <lu...@redhat.com>
11/03/2011 01:23 PM
To
Doug Davis/Raleigh/IBM@IBMUS
cc
deltacloud-dev@incubator.apache.org
Subject
Re: Sketch of CIMI model layer
On Thu, 2011-11-03 at 12:52 -0400, Doug Davis wrote:
> David Lutterkort <lu...@redhat.com> wrote on 11/03/2011 12:36:35 PM:
> ...
> >
> > The issue really isn't so much that there is this small difference
> > between JSON and XML, but that the rules for the JSON/XML
correspondence
> > aren't spelled out. Sections 4.1.9 and 4.1.10 in the WIP talk a little
> > bit about that, but a slightly more formal (structural induction or
so)
> > set of rules would definitely ease the mind of any implementor. BTW,
> > 4.1.10 contradicts the examples later on, like the one on page 37 for
a
> > MachineTemplate - 4.1.10 talks about a wrapper property, whereas the
> > examples don't have one.
>
> can you elaborate on the mismatch between 4.1.10 and MachineTemplate?
> Which part of MachineTemplate is missing a wrapper? I'd like to fix it.
In the WIP[1], section 4.1.10 says that arrays are serialized with a
wrapper property
"volumes" : [
{ "volume": { "href": string },
"attachmentPoint": string,
"protocol": string } + ], ?
Whereas the example on page 39 has
{ "uri": string,
"name": string,
...
"volumes": [
{ "href": string, "attachmentPoint": string, "protocol":
string }, + ], ?
... }
I actually think the second form is a little nicer than the first one;
you'd write 'volumes[0].href' rather than 'volumes[0].volume.href'. For
serializing back to XML, the second form also seems preferrable, since
the serialization only needs to follow two rules
* an array property p is serialized as a sequence of elements
whose name is the singular of the array property name ('volumes'
-> 'volume')
* properties of p[i] that are marked as XML attributes in the
schema are serialized as attributes on the enclosing tag
(<volume href='...'>)
* properties of p[i] that are not marked as XML attributes are
serialized as child elements.
I am sure you can come up with similar rules for the first form, though
they aren't quiteas obvious to me ;)
David
[1]
http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.0a.pdf
Re: Sketch of CIMI model layer
Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2011-11-03 at 12:52 -0400, Doug Davis wrote:
> David Lutterkort <lu...@redhat.com> wrote on 11/03/2011 12:36:35 PM:
> ...
> >
> > The issue really isn't so much that there is this small difference
> > between JSON and XML, but that the rules for the JSON/XML correspondence
> > aren't spelled out. Sections 4.1.9 and 4.1.10 in the WIP talk a little
> > bit about that, but a slightly more formal (structural induction or so)
> > set of rules would definitely ease the mind of any implementor. BTW,
> > 4.1.10 contradicts the examples later on, like the one on page 37 for a
> > MachineTemplate - 4.1.10 talks about a wrapper property, whereas the
> > examples don't have one.
>
> can you elaborate on the mismatch between 4.1.10 and MachineTemplate?
> Which part of MachineTemplate is missing a wrapper? I'd like to fix it.
In the WIP[1], section 4.1.10 says that arrays are serialized with a
wrapper property
"volumes" : [
{ "volume": { "href": string },
"attachmentPoint": string,
"protocol": string } + ], ?
Whereas the example on page 39 has
{ "uri": string,
"name": string,
...
"volumes": [
{ "href": string, "attachmentPoint": string, "protocol": string }, + ], ?
... }
I actually think the second form is a little nicer than the first one;
you'd write 'volumes[0].href' rather than 'volumes[0].volume.href'. For
serializing back to XML, the second form also seems preferrable, since
the serialization only needs to follow two rules
* an array property p is serialized as a sequence of elements
whose name is the singular of the array property name ('volumes'
-> 'volume')
* properties of p[i] that are marked as XML attributes in the
schema are serialized as attributes on the enclosing tag
(<volume href='...'>)
* properties of p[i] that are not marked as XML attributes are
serialized as child elements.
I am sure you can come up with similar rules for the first form, though
they aren't quiteas obvious to me ;)
David
[1] http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.0a.pdf
Re: Sketch of CIMI model layer
Posted by Doug Davis <du...@us.ibm.com>.
David Lutterkort <lu...@redhat.com> wrote on 11/03/2011 12:36:35 PM:
...
>
> The issue really isn't so much that there is this small difference
> between JSON and XML, but that the rules for the JSON/XML correspondence
> aren't spelled out. Sections 4.1.9 and 4.1.10 in the WIP talk a little
> bit about that, but a slightly more formal (structural induction or so)
> set of rules would definitely ease the mind of any implementor. BTW,
> 4.1.10 contradicts the examples later on, like the one on page 37 for a
> MachineTemplate - 4.1.10 talks about a wrapper property, whereas the
> examples don't have one.
can you elaborate on the mismatch between 4.1.10 and MachineTemplate?
Which part of MachineTemplate is missing a wrapper? I'd like to fix it.
-Doug
Re: Sketch of CIMI model layer
Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2011-11-03 at 11:46 +0200, marios@redhat.com wrote:
> On 03/11/11 01:08, lutter@redhat.com wrote:
> > This patch is a first attempt at a CIMI model layer. Such a layer is needed
> > to deal with deserialization (and possibly serialization) of CIMI model
> > objects from (to) XML/JSON, and to give us a place to stash business logic
> > connected to model classes.
> >
> > This patch is solely for discussion, it most definitely will not work; at a
> > minimum there are various 'require' that are missing.
> >
> > There's two things that make such a model layer interesting:
> >
> > * The need to deal with differences between the JSON and XML
> > representation; I believe they are all mechanical, though the CIMI
> > draft does not set out any explicit rules. The main rule is that an
> > array of things in JSON is represented as { 'things': [ .. array .. ] }
> > whereas the XML uses a sequence of <thing/> elements
>
> I recall discussion a while back about whether we need to include
> wrappers for 'array' elements in the xml serialization... i.e.
>
> <volumes>
> <volume href="xs:anyURI"
> attachmentPoint="xs:string" protocol="xs:string" />
> <volume href="xs:anyURI"
> attachmentPoint="xs:string" protocol="xs:string" />
> <volumes/>
>
> (not sure but may have been in relation to mantis #1195). I wonder
> whether this needs to be revisited if it will help to bridge the
> difference between json+xml 'arrays' and help with our deserialization.
No, I don't think it would make much of a difference; with XmlSimple,
the current form without wrappers is actually more convenient. With or
without wrappers there will be a small difference between XML and JSON
handling for arrays.
The issue really isn't so much that there is this small difference
between JSON and XML, but that the rules for the JSON/XML correspondence
aren't spelled out. Sections 4.1.9 and 4.1.10 in the WIP talk a little
bit about that, but a slightly more formal (structural induction or so)
set of rules would definitely ease the mind of any implementor. BTW,
4.1.10 contradicts the examples later on, like the one on page 37 for a
MachineTemplate - 4.1.10 talks about a wrapper property, whereas the
examples don't have one.
In summary: no wrappers is perfectly fine, explicit rules, especially
ones that could be used for JSON <-> XML transformation would be useful.
> The spec doees currently include 'cardinality characters' after items to
> indicate:
>
> "?" (0 or 1)
> "*" (0 or more)
> "+" (1 or more)
>
> e.g. the xml serialization of the volume attribute for machine template:
>
> <volume href="xs:anyURI"
> attachmentPoint="xs:string" protocol="xs:string" /> *
>
> but I don't think that's enough metadata to help us; it does tell us
> that there may be more of this <thing> but we still need to search
> through the xml doc to find all occurrences.
Yes, though it's not a big deal; XmlSimple behaves like that by default,
and even with XPath it would just be a matter of writing
xmldoc.("/machineTemplate/volume")
> > * see if we can really have a Base.to_xml; the two big issues to settle
> > are
> > (1) indicate which attributes go into XML attributes and which ones
> > go into elements, e.g.
> > ref :foo, :xml_attr => true
>
> we could in theory maintain the list of 'entities' though we would need
> to maintain and update it with the spec... - machine, machine template,
> volume, network, job, meter etc ... any <thing> that's not an entity is
> by default an xml attribute.
I believe we need some sort of explicit schema/metadata anyway (and
schema here can really be sth defined in the Ruby code, doesn't have to
be XSD/RNG): when we start a Machine, we take in a MachineTemplate; that
means we need to create some sort of internal representation of a
Machine from scratch. In theory, that could be a hash, or 15 different
variables (@machine_name, @machine_disks, ...) that we pull together in
the output template.
For code sanity and testability though it's much preferrable to have a
class dedicated to representing a Machine, so that we can have some
simple validation logic there (are all mandatory attributes filled ? Do
the things that should be URI's look like URI's ? ...) so that we can
fail early instead of producing invalid output.
Yes, the fact that things need to be updated with the spec will lead to
major gymnastics, and might be an argument for keeping the metadata
external to the code (one set of text files for v1, one for v2), but I
think we should cross that bridge when we get to it. For now, expressing
the metadata with a DSL seems the simplest, and sufficient, solution.
> Having said that, the spec does include
> extensibility mechanisms whereby cloud providers can extend the model by
> defining new resource types, though these are advertised with
> 'EntityMetadata' resource (but this last point is a recommendation only).
Luckily, for this, we only need to produce _an_ implementation of the
spec. How many optional features/extensions ours supports is up in the
air, and for now, we'll just focus on getting the smallest thing that
might possibly work together.
David
Re: Sketch of CIMI model layer
Posted by "marios@redhat.com" <ma...@redhat.com>.
On 03/11/11 01:08, lutter@redhat.com wrote:
> This patch is a first attempt at a CIMI model layer. Such a layer is needed
> to deal with deserialization (and possibly serialization) of CIMI model
> objects from (to) XML/JSON, and to give us a place to stash business logic
> connected to model classes.
>
> This patch is solely for discussion, it most definitely will not work; at a
> minimum there are various 'require' that are missing.
>
> There's two things that make such a model layer interesting:
>
> * The need to deal with differences between the JSON and XML
> representation; I believe they are all mechanical, though the CIMI
> draft does not set out any explicit rules. The main rule is that an
> array of things in JSON is represented as { 'things': [ .. array .. ] }
> whereas the XML uses a sequence of <thing/> elements
I recall discussion a while back about whether we need to include
wrappers for 'array' elements in the xml serialization... i.e.
<volumes>
<volume href="xs:anyURI"
attachmentPoint="xs:string" protocol="xs:string" />
<volume href="xs:anyURI"
attachmentPoint="xs:string" protocol="xs:string" />
<volumes/>
(not sure but may have been in relation to mantis #1195). I wonder
whether this needs to be revisited if it will help to bridge the
difference between json+xml 'arrays' and help with our deserialization.
The spec doees currently include 'cardinality characters' after items to
indicate:
"?" (0 or 1)
"*" (0 or more)
"+" (1 or more)
e.g. the xml serialization of the volume attribute for machine template:
<volume href="xs:anyURI"
attachmentPoint="xs:string" protocol="xs:string" /> *
but I don't think that's enough metadata to help us; it does tell us
that there may be more of this <thing> but we still need to search
through the xml doc to find all occurrences.
> * see if we can really have a Base.to_xml; the two big issues to settle
> are
> (1) indicate which attributes go into XML attributes and which ones
> go into elements, e.g.
> ref :foo, :xml_attr => true
we could in theory maintain the list of 'entities' though we would need
to maintain and update it with the spec... - machine, machine template,
volume, network, job, meter etc ... any <thing> that's not an entity is
by default an xml attribute. Having said that, the spec does include
extensibility mechanisms whereby cloud providers can extend the model by
defining new resource types, though these are advertised with
'EntityMetadata' resource (but this last point is a recommendation only).
>
> (2) for attributes that are not scalars, do (1) recursively; in
> particular, allow associating a class with the members of an
> array
>
> array :things, :class => Thing
>
> where Thing itself is a subclass of Base
>
> * some type specific validation (a 'ref' should look like a URI, we could
> add types to 'scalar' and make sure integers look like integers etc.)
>
> * a unit test or two
>
> Another sidenote: since the CIMI API isn't terribly HTML friendly (e.g.,
> POST's all have XML or JSON bodies), I wonder if the HTML frontend
> shouldn't be a JS app that communicates with the server via JSON only.
>
> David