You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by mf...@redhat.com on 2012/11/29 14:05:38 UTC

Initial support for persistence layer (rev 3)

Hi,

This revision add support for MachineTemplate model in CIMI.

How to use it:

curl -v -X POST -H 'Content-Type: application/json' --user \
  'mockuser:mockpassword' -d @machine_template.json \
  http://localhost:3001/cimi/machine_templates

(machine_template.json is attached).

Then if you want to create a new machine using this MachineTemplate:

curl -v -X POST -H 'Content-Type: application/json' --user \
  'mockuser:mockpassword' -d @machine_from_temp.json \
  http://localhost:3001/cimi/machines

(machine_from_temp.json) is attached)

  -- Michal


Re: Initial support for persistence layer (rev 3)

Posted by Michal Fojtik <mf...@redhat.com>.
On 11/29, mfojtik@redhat.com wrote:

I just noticed that our unit tests are failing with this patchset.
For some (unknown) reason DataMapper cause Rack::Test to explode.

I have patch ready that will prevent use DataMapper while running
under ENV['RACK_ENV'] == 'test'.

  -- Michal

> Hi,
> 
> This revision add support for MachineTemplate model in CIMI.
> 
> How to use it:
> 
> curl -v -X POST -H 'Content-Type: application/json' --user \
>   'mockuser:mockpassword' -d @machine_template.json \
>   http://localhost:3001/cimi/machine_templates
> 
> (machine_template.json is attached).
> 
> Then if you want to create a new machine using this MachineTemplate:
> 
> curl -v -X POST -H 'Content-Type: application/json' --user \
>   'mockuser:mockpassword' -d @machine_from_temp.json \
>   http://localhost:3001/cimi/machines
> 
> (machine_from_temp.json) is attached)
> 
>   -- Michal
> 

-- 
Michal Fojtik <mf...@redhat.com>
Deltacloud API, CloudForms

Re: [PATCH core 4/6] CIMI: Added MachineTemplate database model

Posted by Michal Fojtik <mf...@redhat.com>.
On 11/29, David Lutterkort wrote:
> On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> > From: Michal Fojtik <mf...@redhat.com>
> > 
> > * Added BaseEntity as a parent model for Entity
> > 
> > Signed-off-by: Michal fojtik <mf...@redhat.com>
> 
> ACK. Nice. Though do we need to call it MachineTemplateEntity ? Isn't
> Deltacloud::Database::MachineTemplate enough ? If we do need the extra
> distinction, I suggest we use 'Record' instead of 'Entity', since the
> latter is so overloaded, i.e. call this MachineTemplateRecord

Sure, I renamed it to just MachineTemplate, hope the ruby namespacing works
well :-)

I was a bit afraid about confusion caused by having two classes with the
same name, but not a big deal :-)

  -- Michal


-- 
Michal Fojtik <mf...@redhat.com>
Deltacloud API, CloudForms

Re: [PATCH core 4/6] CIMI: Added MachineTemplate database model

Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> From: Michal Fojtik <mf...@redhat.com>
> 
> * Added BaseEntity as a parent model for Entity
> 
> Signed-off-by: Michal fojtik <mf...@redhat.com>

ACK. Nice. Though do we need to call it MachineTemplateEntity ? Isn't
Deltacloud::Database::MachineTemplate enough ? If we do need the extra
distinction, I suggest we use 'Record' instead of 'Entity', since the
latter is so overloaded, i.e. call this MachineTemplateRecord

David



[PATCH core 4/6] CIMI: Added MachineTemplate database model

Posted by mf...@redhat.com.
From: Michal Fojtik <mf...@redhat.com>

* Added BaseEntity as a parent model for Entity

Signed-off-by: Michal fojtik <mf...@redhat.com>
---
 server/lib/db.rb                  |  1 +
 server/lib/db/entity.rb           | 11 +++++++++--
 server/lib/db/machine_template.rb | 13 +++++++++++++
 server/lib/db/provider.rb         |  1 +
 4 files changed, 24 insertions(+), 2 deletions(-)
 create mode 100644 server/lib/db/machine_template.rb

diff --git a/server/lib/db.rb b/server/lib/db.rb
index e0f28d4..db98709 100644
--- a/server/lib/db.rb
+++ b/server/lib/db.rb
@@ -4,6 +4,7 @@ module Deltacloud
 
   require_relative './db/provider'
   require_relative './db/entity'
+  require_relative './db/machine_template'
 
   DATABASE_LOCATION = ENV['DATABASE_LOCATION'] || "/var/tmp/deltacloud-mock-#{ENV['USER']}/db.sqlite"
 
diff --git a/server/lib/db/entity.rb b/server/lib/db/entity.rb
index b220f9e..c9845a4 100644
--- a/server/lib/db/entity.rb
+++ b/server/lib/db/entity.rb
@@ -1,19 +1,26 @@
 module Deltacloud
   module Database
 
-    class Entity
+    class BaseEntity
       include DataMapper::Resource
 
       belongs_to :provider
 
       property :id, Serial
+      property :type, Discriminator
+
       property :be_kind, String, :required => true # => Machine, MachineImage, ...
       property :be_id, String, :required => true # => Original Machine 'id'
 
-      property :ent_properties, Json
+    end
+
+    class Entity < BaseEntity
+
+      belongs_to :provider
 
       property :name, String
       property :description, String
+      property :ent_properties, Json
 
       def to_hash
         retval = {}
diff --git a/server/lib/db/machine_template.rb b/server/lib/db/machine_template.rb
new file mode 100644
index 0000000..813588d
--- /dev/null
+++ b/server/lib/db/machine_template.rb
@@ -0,0 +1,13 @@
+module Deltacloud
+  module Database
+
+    class MachineTemplateEntity < Entity
+      belongs_to :provider
+
+      property :machine_config, String
+      property :machine_image, String
+
+    end
+
+  end
+end
diff --git a/server/lib/db/provider.rb b/server/lib/db/provider.rb
index 6a761e5..e227a73 100644
--- a/server/lib/db/provider.rb
+++ b/server/lib/db/provider.rb
@@ -9,6 +9,7 @@ module Deltacloud
       property :url, Text
 
       has n, :entities
+      has n, :machine_template_entities
     end
 
   end
-- 
1.8.0


Re: [PATCH core 5/6] CIMI: Added MachineTemplate collection and CRUD operations to model

Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> From: Michal Fojtik <mf...@redhat.com>
> 
> 
> Signed-off-by: Michal fojtik <mf...@redhat.com>
> ---
>  server/lib/cimi/collections/machine_templates.rb | 73 ++++++++++++++++++++++++
>  server/lib/cimi/models/machine_template.rb       | 63 ++++++++++++++++++++
>  2 files changed, 136 insertions(+)
>  create mode 100644 server/lib/cimi/collections/machine_templates.rb

ACK

David



[PATCH core 5/6] CIMI: Added MachineTemplate collection and CRUD operations to model

Posted by mf...@redhat.com.
From: Michal Fojtik <mf...@redhat.com>


Signed-off-by: Michal fojtik <mf...@redhat.com>
---
 server/lib/cimi/collections/machine_templates.rb | 73 ++++++++++++++++++++++++
 server/lib/cimi/models/machine_template.rb       | 63 ++++++++++++++++++++
 2 files changed, 136 insertions(+)
 create mode 100644 server/lib/cimi/collections/machine_templates.rb

diff --git a/server/lib/cimi/collections/machine_templates.rb b/server/lib/cimi/collections/machine_templates.rb
new file mode 100644
index 0000000..17dfbd1
--- /dev/null
+++ b/server/lib/cimi/collections/machine_templates.rb
@@ -0,0 +1,73 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.  The
+# ASF licenses this file to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance with the
+# License.  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+module CIMI::Collections
+  class MachineTemplates < Base
+
+    set :capability, lambda { |t| true }
+
+    collection :machine_templates do
+
+      operation :index do
+        description "List all machine templates"
+        control do
+          machine_templates = MachineTemplate.list(self).filter_by(params['$select'])
+          respond_to do |format|
+            format.xml { machine_templates.to_xml }
+            format.json { machine_templates.to_json }
+          end
+        end
+      end
+
+      operation :show do
+        description "Show specific machine template"
+        control do
+          machine_template = MachineTemplate.find(params[:id], self)
+          respond_to do |format|
+            format.xml { machine_template.to_xml }
+            format.json { machine_template.to_json }
+          end
+        end
+      end
+
+      operation :create do
+        description "Create new machine template"
+        control do
+          if request.content_type.end_with?("json")
+            new_machine_template = MachineTemplate.create_from_json(request.body.read, self)
+          else
+            new_machine_template = MachineTemplate.create_from_xml(request.body.read, self)
+          end
+          status 201 # Created
+          headers 'Location' => new_machine_template.id
+          respond_to do |format|
+            format.json { new_machine_template.to_json }
+            format.xml { new_machine_template.to_xml }
+          end
+        end
+      end
+
+      operation :destroy do
+        description "Delete a specified machine template"
+        control do
+          MachineTemplate.delete!(params[:id], self)
+          no_content_with_status(200)
+        end
+      end
+
+    end
+
+  end
+end
diff --git a/server/lib/cimi/models/machine_template.rb b/server/lib/cimi/models/machine_template.rb
index 7b64b98..fbc6326 100644
--- a/server/lib/cimi/models/machine_template.rb
+++ b/server/lib/cimi/models/machine_template.rb
@@ -40,4 +40,67 @@ class CIMI::Model::MachineTemplate < CIMI::Model::Base
   array :operations do
     scalar :rel, :href
   end
+
+  class << self
+    def find(id, context)
+      if id == :all
+        context.current_db.machine_template_entities.all.map { |t| from_db(t, context) }
+      else
+        template = context.current_db.machine_template_entities.first(:id => id)
+        raise CIMI::Model::NotFound unless template
+        from_db(template, context)
+      end
+    end
+
+    def create_from_json(body, context)
+      json = JSON.parse(body)
+      new_template = context.current_db.machine_template_entities.new(
+        :name => json['name'],
+        :description => json['description'],
+        :machine_config => json['machineConfig']['href'],
+        :machine_image => json['machineImage']['href'],
+        :ent_properties => json['properties'].to_json,
+        :be_kind => 'machine_template',
+        :be_id => ''
+      )
+      new_template.save!
+      from_db(new_template, context)
+    end
+
+    def create_from_xml(body, context)
+      xml = XmlSimple.xml_in(body)
+      new_template = context.current_db.machine_template_entities.new(
+        :name => xml['name'].first,
+        :description => xml['description'].first,
+        :machine_config => xml['machineConfig'].first['href'],
+        :machine_image => xml['machineImage'].first['href'],
+        :ent_properties => xml['properties'].first.to_json,
+        :be_kind => 'machine_template',
+        :be_id => ''
+      )
+      new_template.save!
+      from_db(new_template, context)
+    end
+
+    def delete!(id, context)
+      context.current_db.machine_template_entities.first(:id => id).destroy
+    end
+
+    private
+
+    def from_db(model, context)
+      self.new(
+        :id => context.machine_template_url(model.id),
+        :name => model.name,
+        :description => model.description,
+        :machine_config => { :href => model.machine_config },
+        :machine_image => { :href => model.machine_image },
+        :property => model.ent_properties,
+        :operations => [
+          { :href => context.destroy_machine_template_url(model.id), :rel => 'http://schemas.dmtf.org/cimi/1/action/delete' }
+        ]
+      )
+    end
+  end
+
 end
-- 
1.8.0


Re: [PATCH core 6/6] CIMI: Added support for creating Machine using MachineTemplate

Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> From: Michal Fojtik <mf...@redhat.com>
> 
> 
> Signed-off-by: Michal fojtik <mf...@redhat.com>
> ---
>  server/lib/cimi/models/machine.rb | 20 +++++++++++++++++---
>  1 file changed, 17 insertions(+), 3 deletions(-)

ACK

David



[PATCH core 6/6] CIMI: Added support for creating Machine using MachineTemplate

Posted by mf...@redhat.com.
From: Michal Fojtik <mf...@redhat.com>


Signed-off-by: Michal fojtik <mf...@redhat.com>
---
 server/lib/cimi/models/machine.rb | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/server/lib/cimi/models/machine.rb b/server/lib/cimi/models/machine.rb
index 07c5a17..f6d19b5 100644
--- a/server/lib/cimi/models/machine.rb
+++ b/server/lib/cimi/models/machine.rb
@@ -48,8 +48,14 @@ class CIMI::Model::Machine < CIMI::Model::Base
   def self.create_from_json(body, context)
     json = JSON.parse(body)
     machine_template = json['machineTemplate']
-    hardware_profile_id = machine_template['machineConfig']["href"].split('/').last
-    image_id = machine_template['machineImage']["href"].split('/').last
+    if !machine_template['href'].nil?
+      template = context.current_db.machine_template_entities.first(:id => machine_template['href'].split('/').last)
+      hardware_profile_id = template.machine_config.split('/').last
+      image_id = template.machine_image.split('/').last
+    else
+      hardware_profile_id = machine_template['machineConfig']["href"].split('/').last
+      image_id = machine_template['machineImage']["href"].split('/').last
+    end
     additional_params = {}
     additional_params[:name] = json['name'] if json['name']
     if machine_template.has_key? 'credential'
@@ -67,7 +73,15 @@ class CIMI::Model::Machine < CIMI::Model::Base
 
   def self.create_from_xml(body, context)
     xml = XmlSimple.xml_in(body)
-    machine_template = xml['machineTemplate'][0]
+    if !xml['machineTemplate']['href'].nil?
+      template = context.current_db.machine_template_entities.first(:id => xml['machineTemplate']['href'].split('/').last)
+      hardware_profile_id = template.machine_config.split('/').last
+      image_id = template.machine_image.split('/').last
+    else
+      machine_template = xml['machineTemplate'][0]
+      hardware_profile_id = machine_template['machineConfig']["href"].split('/').last
+      image_id = machine_template['machineImage']["href"].split('/').last
+    end
     hardware_profile_id = machine_template['machineConfig'][0]["href"].split('/').last
     image_id = machine_template['machineImage'][0]["href"].split('/').last
     additional_params = {}
-- 
1.8.0


Re: [PATCH core 2/6] Core: Require "json/pure" instead of buggy "json" (DTACLOUD-382)

Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> From: Michal Fojtik <mf...@redhat.com>
> 
> 
> Signed-off-by: Michal fojtik <mf...@redhat.com>
> ---
>  server/lib/cimi/server.rb                             | 1 -
>  server/lib/deltacloud/drivers/gogrid/gogrid_client.rb | 1 -
>  server/lib/deltacloud/drivers/sbc/sbc_client.rb       | 1 -
>  server/lib/deltacloud/server.rb                       | 1 -
>  server/lib/deltacloud_rack.rb                         | 1 +
>  5 files changed, 1 insertion(+), 4 deletions(-)

ACK



[PATCH core 2/6] Core: Require "json/pure" instead of buggy "json" (DTACLOUD-382)

Posted by mf...@redhat.com.
From: Michal Fojtik <mf...@redhat.com>


Signed-off-by: Michal fojtik <mf...@redhat.com>
---
 server/lib/cimi/server.rb                             | 1 -
 server/lib/deltacloud/drivers/gogrid/gogrid_client.rb | 1 -
 server/lib/deltacloud/drivers/sbc/sbc_client.rb       | 1 -
 server/lib/deltacloud/server.rb                       | 1 -
 server/lib/deltacloud_rack.rb                         | 1 +
 5 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/server/lib/cimi/server.rb b/server/lib/cimi/server.rb
index 80e6200..9462f6f 100644
--- a/server/lib/cimi/server.rb
+++ b/server/lib/cimi/server.rb
@@ -15,7 +15,6 @@
 
 require 'rubygems'
 require 'crack'
-require 'json'
 require 'yaml'
 require 'haml'
 require 'sinatra/base'
diff --git a/server/lib/deltacloud/drivers/gogrid/gogrid_client.rb b/server/lib/deltacloud/drivers/gogrid/gogrid_client.rb
index 77fba97..31770e9 100644
--- a/server/lib/deltacloud/drivers/gogrid/gogrid_client.rb
+++ b/server/lib/deltacloud/drivers/gogrid/gogrid_client.rb
@@ -17,7 +17,6 @@
 require 'digest/md5'
 require 'cgi'
 require 'open-uri'
-require 'json'
 
 module Kernel
   def suppress_warnings
diff --git a/server/lib/deltacloud/drivers/sbc/sbc_client.rb b/server/lib/deltacloud/drivers/sbc/sbc_client.rb
index 97e89ca..0316467 100644
--- a/server/lib/deltacloud/drivers/sbc/sbc_client.rb
+++ b/server/lib/deltacloud/drivers/sbc/sbc_client.rb
@@ -16,7 +16,6 @@
 
 require 'base64'
 require 'net/https'
-require 'json'
 require 'digest/md5'
 
 module Deltacloud
diff --git a/server/lib/deltacloud/server.rb b/server/lib/deltacloud/server.rb
index 53249f3..7510dba 100644
--- a/server/lib/deltacloud/server.rb
+++ b/server/lib/deltacloud/server.rb
@@ -15,7 +15,6 @@
 
 require 'rubygems'
 require 'crack'
-require 'json'
 require 'yaml'
 require 'haml'
 require 'sinatra/base'
diff --git a/server/lib/deltacloud_rack.rb b/server/lib/deltacloud_rack.rb
index 4999083..b918566 100644
--- a/server/lib/deltacloud_rack.rb
+++ b/server/lib/deltacloud_rack.rb
@@ -14,6 +14,7 @@
 # under the License.
 
 require 'require_relative' if RUBY_VERSION < '1.9'
+require 'json/pure'
 
 require_relative './deltacloud/core_ext'
 require_relative './sinatra/rack_logger'
-- 
1.8.0


Re: [PATCH core 3/6] CIMI: Fix the case when stored attributes are XML

Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> From: Michal Fojtik <mf...@redhat.com>
> 
> In XmlSimple the values of elements are actually Array.
> In that case we need to do an extra step to extract the correct
> String value that is being saved to the database.
> 
> Signed-off-by: Michal fojtik <mf...@redhat.com>

ACK, but it should really be distributed through some of the other
patches; e.g. the change to dependencies.rb seems to belong in 2/6, the
change to base.rb belongs into 1/6. The change to volume.rb should be a
separate patch, with a very different description.

David



[PATCH core 3/6] CIMI: Fix the case when stored attributes are XML

Posted by mf...@redhat.com.
From: Michal Fojtik <mf...@redhat.com>

In XmlSimple the values of elements are actually Array.
In that case we need to do an extra step to extract the correct
String value that is being saved to the database.

Signed-off-by: Michal fojtik <mf...@redhat.com>
---
 server/lib/cimi/dependencies.rb  |  1 -
 server/lib/cimi/models/base.rb   | 15 ++++++++++----
 server/lib/cimi/models/volume.rb | 44 +++++++++++++++++++++++++++++++++++-----
 3 files changed, 50 insertions(+), 10 deletions(-)

diff --git a/server/lib/cimi/dependencies.rb b/server/lib/cimi/dependencies.rb
index d02ff30..6f0000c 100644
--- a/server/lib/cimi/dependencies.rb
+++ b/server/lib/cimi/dependencies.rb
@@ -39,7 +39,6 @@ require 'deltacloud/models/load_balancer'
 require 'deltacloud/models/firewall'
 require 'deltacloud/models/firewall_rule'
 
-require 'json'
 require 'sinatra/rack_accept'
 require 'sinatra/rack_cimi'
 require 'sinatra/static_assets'
diff --git a/server/lib/cimi/models/base.rb b/server/lib/cimi/models/base.rb
index a5d8f34..2b13367 100644
--- a/server/lib/cimi/models/base.rb
+++ b/server/lib/cimi/models/base.rb
@@ -14,7 +14,6 @@
 # under the License.
 
 require 'xmlsimple'
-require 'json'
 
 # The base class for any CIMI object that we either read from a request or
 # write as a response. This class handles serializing/deserializing XML and
@@ -289,10 +288,18 @@ class CIMI::Model::Base < CIMI::Model::Resource
   class << self
     def store_attributes_for(context, entity, attrs={})
       stored_attributes = {}
-      stored_attributes[:description] = attrs['description'] if attrs['description']
-      stored_attributes[:name] = attrs['name'] if attrs['name']
-      stored_attributes[:ent_properties] = attrs['properties'].to_json if attrs['properties']
+      stored_attributes[:description] = extract_attribute_value('description', attrs) if attrs['description']
+      stored_attributes[:name] = extract_attribute_value('name', attrs) if attrs['name']
+      stored_attributes[:ent_properties] = extract_attribute_value('properties', attrs).to_json if attrs['properties']
       context.store_attributes_for(entity, stored_attributes)
     end
+
+    # In XML serialization the values stored in attrs are arrays, dues to
+    # XmlSimple. This method will help extract values from them
+    #
+    def extract_attribute_value(name, attrs={})
+      return unless attrs[name]
+      attrs[name].is_a?(Array) ? attrs[name].first : attrs[name]
+    end
   end
 end
diff --git a/server/lib/cimi/models/volume.rb b/server/lib/cimi/models/volume.rb
index d777cd0..b799f26 100644
--- a/server/lib/cimi/models/volume.rb
+++ b/server/lib/cimi/models/volume.rb
@@ -53,7 +53,7 @@ class CIMI::Model::Volume < CIMI::Model::Base
     volume_config_id = json["volumeTemplate"]["volumeConfig"]["href"].split("/").last
     volume_image_id = (json["volumeTemplate"].has_key?("volumeImage") ?
                 json["volumeTemplate"]["volumeImage"]["href"].split("/").last  : nil)
-    create_volume({:volume_config_id=>volume_config_id, :volume_image_id=>volume_image_id}, context)
+    create_volume({:volume_config_id=>volume_config_id, :volume_image_id=>volume_image_id}, json, context)
   end
 
   def self.create_from_xml(xml_in, context)
@@ -61,25 +61,45 @@ class CIMI::Model::Volume < CIMI::Model::Base
     volume_config_id = xml["volumeTemplate"][0]["volumeConfig"][0]["href"].split("/").last
     volume_image_id = (xml["volumeTemplate"][0].has_key?("volumeImage") ?
              xml["volumeTemplate"][0]["volumeImage"][0]["href"].split("/").last  : nil)
-    create_volume({:volume_config_id=>volume_config_id, :volume_image_id=>volume_image_id}, context)
+    create_volume({:volume_config_id=>volume_config_id, :volume_image_id=>volume_image_id}, xml, context)
   end
 
   def self.delete!(id, context)
     context.driver.destroy_storage_volume(context.credentials, {:id=>id} )
+    delete_attributes_for(Volume.new(:id => id))
+  end
+
+  def self.find_to_attach_from_json(json_in, context)
+    json = JSON.parse(json_in)
+    json["volumes"].map{|v| {:volume=>self.find(v["volume"]["href"].split("/volumes/").last, context),
+                             :attachment_point=>v["attachmentPoint"]  }}
+  end
+
+  def self.find_to_attach_from_xml(xml_in, context)
+    xml = XmlSimple.xml_in(xml_in)
+    xml["volume"].map{|v| {:volume => self.find(v["href"].split("/volumes/").last, context),
+                           :attachment_point=>v["attachmentPoint"] }}
+  end
+
+  def to_entity
+    'volume'
   end
 
   private
 
-  def self.create_volume(params, context)
+  def self.create_volume(params, data, context)
     volume_config = CIMI::Model::VolumeConfiguration.find(params[:volume_config_id], context)
     opts = {:capacity=>context.from_kibibyte(volume_config.capacity, "GB"), :snapshot_id=>params[:volume_image_id] }
     storage_volume = context.driver.create_storage_volume(context.credentials, opts)
+    store_attributes_for(context, storage_volume, data)
     from_storage_volume(storage_volume, context)
   end
 
   def self.from_storage_volume(volume, context)
-    self.new( { :name => volume.id,
-                :description => volume.id,
+    stored_attributes = context.load_attributes_for(volume)
+    self.new( { :name => stored_attributes[:name] || volume.id,
+                :description => stored_attributes[:description] || 'Description of Volume',
+                :property => stored_attributes[:property],
                 :created => volume.created.nil? ? nil : Time.parse(volume.created).xmlschema,
                 :id => context.volume_url(volume.id),
                 :capacity => context.to_kibibyte(volume.capacity, 'GB'),
@@ -91,4 +111,18 @@ class CIMI::Model::Volume < CIMI::Model::Base
             } )
   end
 
+  def self.collection_for_instance(instance_id, context)
+    instance = context.driver.instance(context.credentials, :id => instance_id)
+    volumes = instance.storage_volumes.map do |mappings|
+      mappings.keys.map { |volume_id| from_storage_volume(context.driver.storage_volume(context.credentials, :id => volume_id), context) }
+    end.flatten
+    CIMI::Model::VolumeCollection.new(
+      :id => context.url("/machines/#{instance_id}/volumes"),
+      :name => 'default',
+      :count => volumes.size,
+      :description => "Volume collection for Machine #{instance_id}",
+      :entries => volumes
+    )
+  end
+
 end
-- 
1.8.0


Re: [PATCH core 1/6] CIMI: Initial support for persisting attributes in database

Posted by David Lutterkort <lu...@redhat.com>.
On Fri, 2012-11-30 at 10:10 +0100, Michal Fojtik wrote:
> On 11/29, David Lutterkort wrote:
> > 
> > > +      def current_provider
> > > +        Thread.current[:provider] || ENV['API_PROVIDER'] || 'default'
> > > +      end
> > 
> > Ugh .. this introduces nasty dependencies on how we are running things.
> > It might be time to write the driver/provider into every
> > Deltacloud::Model (or CIMI::Model) object.
> 
> What you mean with nasty dependencies? We use this in views too
> (api.xml.haml, etc). I think the best place for this method will be
> 'drivers.rb.

For views, I have an easier time with depending on global variables like
Thread.current or ENV, as they are really only useful after you've gone
through the whole machinery of handling a request. But if you also
depend on global vars deep down in code that stores in the DB, it feels
we've introduced a dependency on how request handling is done, and what
it sets up, into code that should be far removed from request handling.

I am not sure if my model suggestion will be enough to clean up this
dependeny, but it's worth thinking about that some more.

David



Re: [PATCH core 1/6] CIMI: Initial support for persisting attributes in database

Posted by Michal Fojtik <mf...@redhat.com>.
On 11/29, David Lutterkort wrote:
> Hi Michal,
> 
> ACK. Some questions:
> 
> On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> > diff --git a/server/config.ru b/server/config.ru
> > index efed62f..406c769 100644
> > --- a/server/config.ru
> > +++ b/server/config.ru
> > @@ -54,6 +54,7 @@ end
> >  #       different root_url's
> >  #
> >  frontends.each do |frontend|
> > +  frontend = frontend.strip
> 
> This should go into a separate patch - presumably it's to fix user
> typos ?

Right. Sorry for that, I really need to get more familiar with git add :/

> > diff --git a/server/lib/cimi/models/base.rb b/server/lib/cimi/models/base.rb
> > index 6a73cf6..a5d8f34 100644
> > --- a/server/lib/cimi/models/base.rb
> > +++ b/server/lib/cimi/models/base.rb
> > @@ -285,4 +285,14 @@ class CIMI::Model::Base < CIMI::Model::Resource
> >      end
> >      self.class.new(attrs)
> >    end
> > +
> > +  class << self
> > +    def store_attributes_for(context, entity, attrs={})
> > +      stored_attributes = {}
> > +      stored_attributes[:description] = attrs['description'] if attrs['description']
> > +      stored_attributes[:name] = attrs['name'] if attrs['name']
> > +      stored_attributes[:ent_properties] = attrs['properties'].to_json if attrs['properties']
> > +      context.store_attributes_for(entity, stored_attributes)
> 
> Why do we need the context here at all ? If we store in the database on
> the level of CIMI model objects, rather than backend (driver) objects
> there's no need for that. I don't like the idea that we need to pass the
> context/driver this deep down, and that the responsibility for
> storing/retrieving to/from the DB is distributed this widely (e.g., in
> Machine.delete!).

Good point thanks! I used context to generate 'url' in one of the previous
revisions. Currently there is no point in doing that. I will remove this
right after DMTF meeting :-)

> Please don't address this comment just yet, we'll do that after the DMTF
> meeting.
> 
> > @@ -120,10 +129,16 @@ class CIMI::Model::Machine < CIMI::Model::Base
> >    def self.from_instance(instance, context)
> >      cpu =  memory = (instance.instance_profile.id == "opaque")? "n/a" : nil
> >      machine_conf = CIMI::Model::MachineConfiguration.find(instance.instance_profile.name, context)
> > +    stored_attributes = context.load_attributes_for(instance)
> > +    if stored_attributes[:property]
> > +      stored_attributes[:property].merge!(convert_instance_properties(instance, context))
> > +    else
> > +      stored_attributes[:property] = convert_instance_properties(instance, context)
> > +    end
> 
> How about
> 
>         stored_attributes = context.load_attributes_for(instance) || {}
>         stored_attributes[:property].merge!(convert_instance_properties(instance, context))

ACK. Will add fix it.

> > @@ -131,8 +146,8 @@ class CIMI::Model::Machine < CIMI::Model::Base
> >        :disks => { :href => context.machine_url(instance.id)+"/disks"},
> >        :volumes => { :href=>context.machine_url(instance.id)+"/volumes"},
> >        :operations => convert_instance_actions(instance, context),
> > -      :property => convert_instance_properties(instance, context)
> > -    }
> > +      :property => stored_attributes
> > +    }.merge(stored_attributes)
> 
> The ':property => ... ' line doesn't seem to belong there. The :merge
> should already do the right thing.
> 
> > diff --git a/server/lib/db.rb b/server/lib/db.rb
> > new file mode 100644
> > index 0000000..e0f28d4
> > --- /dev/null
> > +++ b/server/lib/db.rb
> > @@ -0,0 +1,55 @@
> > +module Deltacloud
> > +
> > +  require 'data_mapper'
> > +
> > +  require_relative './db/provider'
> > +  require_relative './db/entity'
> > +
> > +  DATABASE_LOCATION = ENV['DATABASE_LOCATION'] || "/var/tmp/deltacloud-mock-#{ENV['USER']}/db.sqlite"
> > +
> > +  def self.initialize_database
> > +    DataMapper::Logger.new($stdout, :debug)
> > +    DataMapper::setup(:default, "sqlite://#{DATABASE_LOCATION}")
> > +    DataMapper::finalize
> > +    DataMapper::auto_upgrade!
> > +  end
> > +
> > +  module Helpers
> > +    module Database
> 
> Why doesn't this live in lib/deltacloud/helpers ?

I tried to separate all database stuff into one file, but you're right this
need to go to lib/cimi/helpers.

> 
> > +      include Deltacloud::Database
> 
> How do I find that when I first look at the code ? (It's not in
> lib/deltacloud/database.rb)

This is here just because I don't want to write Deltacloud::Database::Entity,
but just Entity. We can safely remove that.

> 
> > +      def current_provider
> > +        Thread.current[:provider] || ENV['API_PROVIDER'] || 'default'
> > +      end
> 
> Ugh .. this introduces nasty dependencies on how we are running things.
> It might be time to write the driver/provider into every
> Deltacloud::Model (or CIMI::Model) object.

What you mean with nasty dependencies? We use this in views too
(api.xml.haml, etc). I think the best place for this method will be
'drivers.rb.

> 
> Again, not something to fix right now, but ultimately, I'd like to get
> to a point where one set of our model objects cleanly encapsulate
> persistence.
> 
> David
> 
> 

-- 
Michal Fojtik <mf...@redhat.com>
Deltacloud API, CloudForms

Re: [PATCH core 1/6] CIMI: Initial support for persisting attributes in database

Posted by David Lutterkort <lu...@redhat.com>.
Hi Michal,

ACK. Some questions:

On Thu, 2012-11-29 at 14:05 +0100, mfojtik@redhat.com wrote:
> diff --git a/server/config.ru b/server/config.ru
> index efed62f..406c769 100644
> --- a/server/config.ru
> +++ b/server/config.ru
> @@ -54,6 +54,7 @@ end
>  #       different root_url's
>  #
>  frontends.each do |frontend|
> +  frontend = frontend.strip

This should go into a separate patch - presumably it's to fix user
typos ?

> diff --git a/server/lib/cimi/models/base.rb b/server/lib/cimi/models/base.rb
> index 6a73cf6..a5d8f34 100644
> --- a/server/lib/cimi/models/base.rb
> +++ b/server/lib/cimi/models/base.rb
> @@ -285,4 +285,14 @@ class CIMI::Model::Base < CIMI::Model::Resource
>      end
>      self.class.new(attrs)
>    end
> +
> +  class << self
> +    def store_attributes_for(context, entity, attrs={})
> +      stored_attributes = {}
> +      stored_attributes[:description] = attrs['description'] if attrs['description']
> +      stored_attributes[:name] = attrs['name'] if attrs['name']
> +      stored_attributes[:ent_properties] = attrs['properties'].to_json if attrs['properties']
> +      context.store_attributes_for(entity, stored_attributes)

Why do we need the context here at all ? If we store in the database on
the level of CIMI model objects, rather than backend (driver) objects
there's no need for that. I don't like the idea that we need to pass the
context/driver this deep down, and that the responsibility for
storing/retrieving to/from the DB is distributed this widely (e.g., in
Machine.delete!).

Please don't address this comment just yet, we'll do that after the DMTF
meeting.

> @@ -120,10 +129,16 @@ class CIMI::Model::Machine < CIMI::Model::Base
>    def self.from_instance(instance, context)
>      cpu =  memory = (instance.instance_profile.id == "opaque")? "n/a" : nil
>      machine_conf = CIMI::Model::MachineConfiguration.find(instance.instance_profile.name, context)
> +    stored_attributes = context.load_attributes_for(instance)
> +    if stored_attributes[:property]
> +      stored_attributes[:property].merge!(convert_instance_properties(instance, context))
> +    else
> +      stored_attributes[:property] = convert_instance_properties(instance, context)
> +    end

How about

        stored_attributes = context.load_attributes_for(instance) || {}
        stored_attributes[:property].merge!(convert_instance_properties(instance, context))

> @@ -131,8 +146,8 @@ class CIMI::Model::Machine < CIMI::Model::Base
>        :disks => { :href => context.machine_url(instance.id)+"/disks"},
>        :volumes => { :href=>context.machine_url(instance.id)+"/volumes"},
>        :operations => convert_instance_actions(instance, context),
> -      :property => convert_instance_properties(instance, context)
> -    }
> +      :property => stored_attributes
> +    }.merge(stored_attributes)

The ':property => ... ' line doesn't seem to belong there. The :merge
should already do the right thing.

> diff --git a/server/lib/db.rb b/server/lib/db.rb
> new file mode 100644
> index 0000000..e0f28d4
> --- /dev/null
> +++ b/server/lib/db.rb
> @@ -0,0 +1,55 @@
> +module Deltacloud
> +
> +  require 'data_mapper'
> +
> +  require_relative './db/provider'
> +  require_relative './db/entity'
> +
> +  DATABASE_LOCATION = ENV['DATABASE_LOCATION'] || "/var/tmp/deltacloud-mock-#{ENV['USER']}/db.sqlite"
> +
> +  def self.initialize_database
> +    DataMapper::Logger.new($stdout, :debug)
> +    DataMapper::setup(:default, "sqlite://#{DATABASE_LOCATION}")
> +    DataMapper::finalize
> +    DataMapper::auto_upgrade!
> +  end
> +
> +  module Helpers
> +    module Database

Why doesn't this live in lib/deltacloud/helpers ?

> +      include Deltacloud::Database

How do I find that when I first look at the code ? (It's not in
lib/deltacloud/database.rb)

> +      def current_provider
> +        Thread.current[:provider] || ENV['API_PROVIDER'] || 'default'
> +      end

Ugh .. this introduces nasty dependencies on how we are running things.
It might be time to write the driver/provider into every
Deltacloud::Model (or CIMI::Model) object.

Again, not something to fix right now, but ultimately, I'd like to get
to a point where one set of our model objects cleanly encapsulate
persistence.

David



[PATCH core 1/6] CIMI: Initial support for persisting attributes in database

Posted by mf...@redhat.com.
From: Michal Fojtik <mf...@redhat.com>

* DataMapper is required for CIMI from this commit.
* This patch support storing of :name, :description and :properties.

Signed-off-by: Michal fojtik <mf...@redhat.com>
---
 server/config.ru                             |  1 +
 server/deltacloud-core.gemspec               |  3 ++
 server/lib/cimi/collections/base.rb          |  1 +
 server/lib/cimi/helpers.rb                   |  1 +
 server/lib/cimi/models/base.rb               | 10 +++++
 server/lib/cimi/models/machine.rb            | 21 +++++++++--
 server/lib/db.rb                             | 55 ++++++++++++++++++++++++++++
 server/lib/db/entity.rb                      | 29 +++++++++++++++
 server/lib/db/provider.rb                    | 15 ++++++++
 server/lib/deltacloud/drivers/base_driver.rb |  4 ++
 server/lib/deltacloud/models/base_model.rb   |  4 ++
 11 files changed, 141 insertions(+), 3 deletions(-)
 create mode 100644 server/lib/db.rb
 create mode 100644 server/lib/db/entity.rb
 create mode 100644 server/lib/db/provider.rb

diff --git a/server/config.ru b/server/config.ru
index efed62f..406c769 100644
--- a/server/config.ru
+++ b/server/config.ru
@@ -54,6 +54,7 @@ end
 #       different root_url's
 #
 frontends.each do |frontend|
+  frontend = frontend.strip
   if Deltacloud[frontend.to_sym].nil?
     puts "ERROR: Unknown frontend (#{frontend}). Valid values are 'deltacloud,cimi,ec2'"
     exit(1)
diff --git a/server/deltacloud-core.gemspec b/server/deltacloud-core.gemspec
index cb9a8c8..0b60396 100644
--- a/server/deltacloud-core.gemspec
+++ b/server/deltacloud-core.gemspec
@@ -79,6 +79,9 @@ Gem::Specification.new do |s|
   s.add_dependency('nokogiri', '>= 1.4.3')
   s.add_dependency('require_relative') if RUBY_VERSION < '1.9'
 
+  s.add_dependency('data_mapper')
+  s.add_dependency('dm-sqlite-adapter')
+
   # dependencies for various cloud providers:
 
   # RHEV-M and oVirt
diff --git a/server/lib/cimi/collections/base.rb b/server/lib/cimi/collections/base.rb
index a78b78a..30c595b 100644
--- a/server/lib/cimi/collections/base.rb
+++ b/server/lib/cimi/collections/base.rb
@@ -23,6 +23,7 @@ module CIMI::Collections
     include CIMI::Model
 
     helpers Deltacloud::Helpers::Drivers
+    helpers Deltacloud::Helpers::Database
     helpers Sinatra::AuthHelper
     helpers Sinatra::Rabbit::URLHelper
     helpers Deltacloud::Helpers::Application
diff --git a/server/lib/cimi/helpers.rb b/server/lib/cimi/helpers.rb
index 9d069b1..c2f2461 100644
--- a/server/lib/cimi/helpers.rb
+++ b/server/lib/cimi/helpers.rb
@@ -26,6 +26,7 @@ end
 # Declare namespace for CIMI models
 #
 
+require_relative '../db'
 require_relative '../deltacloud/drivers'
 require_relative '../deltacloud/models'
 require_relative '../deltacloud/helpers/driver_helper'
diff --git a/server/lib/cimi/models/base.rb b/server/lib/cimi/models/base.rb
index 6a73cf6..a5d8f34 100644
--- a/server/lib/cimi/models/base.rb
+++ b/server/lib/cimi/models/base.rb
@@ -285,4 +285,14 @@ class CIMI::Model::Base < CIMI::Model::Resource
     end
     self.class.new(attrs)
   end
+
+  class << self
+    def store_attributes_for(context, entity, attrs={})
+      stored_attributes = {}
+      stored_attributes[:description] = attrs['description'] if attrs['description']
+      stored_attributes[:name] = attrs['name'] if attrs['name']
+      stored_attributes[:ent_properties] = attrs['properties'].to_json if attrs['properties']
+      context.store_attributes_for(entity, stored_attributes)
+    end
+  end
 end
diff --git a/server/lib/cimi/models/machine.rb b/server/lib/cimi/models/machine.rb
index 21e0081..07c5a17 100644
--- a/server/lib/cimi/models/machine.rb
+++ b/server/lib/cimi/models/machine.rb
@@ -58,6 +58,10 @@ class CIMI::Model::Machine < CIMI::Model::Base
     instance = context.driver.create_instance(context.credentials, image_id, {
       :hwp_id => hardware_profile_id
     }.merge(additional_params))
+
+    # Store attributes that are not supported by the backend cloud to local
+    # database:
+    store_attributes_for(context, instance, json)
     from_instance(instance, context)
   end
 
@@ -74,6 +78,10 @@ class CIMI::Model::Machine < CIMI::Model::Base
     instance = context.driver.create_instance(context.credentials, image_id, {
       :hwp_id => hardware_profile_id
     }.merge(additional_params))
+
+    # Store attributes that are not supported by the backend cloud to local
+    # database:
+    store_attributes_for(context, instance, xml)
     from_instance(instance, context)
   end
 
@@ -90,6 +98,7 @@ class CIMI::Model::Machine < CIMI::Model::Base
   end
 
   def self.delete!(id, context)
+    context.delete_attributes_for Instance.new(:id => id)
     context.driver.destroy_instance(context.credentials, id)
   end
 
@@ -120,10 +129,16 @@ class CIMI::Model::Machine < CIMI::Model::Base
   def self.from_instance(instance, context)
     cpu =  memory = (instance.instance_profile.id == "opaque")? "n/a" : nil
     machine_conf = CIMI::Model::MachineConfiguration.find(instance.instance_profile.name, context)
+    stored_attributes = context.load_attributes_for(instance)
+    if stored_attributes[:property]
+      stored_attributes[:property].merge!(convert_instance_properties(instance, context))
+    else
+      stored_attributes[:property] = convert_instance_properties(instance, context)
+    end
     machine_spec = {
       :name => instance.name,
-      :description => "Instance #{instance.name}",
       :created => instance.launch_time.nil? ? Time.now.xmlschema : Time.parse(instance.launch_time.to_s).xmlschema,
+      :description => "No description set for Machine #{instance.name}",
       :id => context.machine_url(instance.id),
       :state => convert_instance_state(instance.state),
       :cpu => cpu || convert_instance_cpu(instance.instance_profile, context),
@@ -131,8 +146,8 @@ class CIMI::Model::Machine < CIMI::Model::Base
       :disks => { :href => context.machine_url(instance.id)+"/disks"},
       :volumes => { :href=>context.machine_url(instance.id)+"/volumes"},
       :operations => convert_instance_actions(instance, context),
-      :property => convert_instance_properties(instance, context)
-    }
+      :property => stored_attributes
+    }.merge(stored_attributes)
     if context.expand? :disks
       machine_spec[:disks] = CIMI::Model::Disk.find(instance, machine_conf, context, :all)
     end
diff --git a/server/lib/db.rb b/server/lib/db.rb
new file mode 100644
index 0000000..e0f28d4
--- /dev/null
+++ b/server/lib/db.rb
@@ -0,0 +1,55 @@
+module Deltacloud
+
+  require 'data_mapper'
+
+  require_relative './db/provider'
+  require_relative './db/entity'
+
+  DATABASE_LOCATION = ENV['DATABASE_LOCATION'] || "/var/tmp/deltacloud-mock-#{ENV['USER']}/db.sqlite"
+
+  def self.initialize_database
+    DataMapper::Logger.new($stdout, :debug)
+    DataMapper::setup(:default, "sqlite://#{DATABASE_LOCATION}")
+    DataMapper::finalize
+    DataMapper::auto_upgrade!
+  end
+
+  module Helpers
+    module Database
+      include Deltacloud::Database
+
+      def store_attributes_for(model, values={})
+        return if model.nil? or values.empty?
+        current_db.entities.first_or_create(:be_kind => model.to_entity, :be_id => model.id).update(values)
+      end
+
+      def load_attributes_for(model)
+        entity = get_entity(model)
+        entity.nil? ? {} : entity.to_hash
+      end
+
+      def delete_attributes_for(model)
+        get_entity(model).destroy
+      end
+
+      def get_entity(model)
+        current_db.entities.first(:be_kind => model.to_entity, :be_id => model.id)
+      end
+
+      def current_provider
+        Thread.current[:provider] || ENV['API_PROVIDER'] || 'default'
+      end
+
+      # This method allows to store things into database based on current driver
+      # and provider.
+      #
+      def current_db
+        Provider.first_or_create(:driver => driver_symbol.to_s, :url => current_provider)
+      end
+
+    end
+  end
+
+end
+
+Deltacloud::initialize_database
diff --git a/server/lib/db/entity.rb b/server/lib/db/entity.rb
new file mode 100644
index 0000000..b220f9e
--- /dev/null
+++ b/server/lib/db/entity.rb
@@ -0,0 +1,29 @@
+module Deltacloud
+  module Database
+
+    class Entity
+      include DataMapper::Resource
+
+      belongs_to :provider
+
+      property :id, Serial
+      property :be_kind, String, :required => true # => Machine, MachineImage, ...
+      property :be_id, String, :required => true # => Original Machine 'id'
+
+      property :ent_properties, Json
+
+      property :name, String
+      property :description, String
+
+      def to_hash
+        retval = {}
+        retval.merge!(:name => self.name) if !self.name.nil?
+        retval.merge!(:description => self.description) if !self.description.nil?
+        retval.merge!(:property => self.ent_properties) if !self.ent_properties.nil?
+        retval
+      end
+
+    end
+
+  end
+end
diff --git a/server/lib/db/provider.rb b/server/lib/db/provider.rb
new file mode 100644
index 0000000..6a761e5
--- /dev/null
+++ b/server/lib/db/provider.rb
@@ -0,0 +1,15 @@
+module Deltacloud
+  module Database
+
+    class Provider
+      include DataMapper::Resource
+
+      property :id, Serial
+      property :driver, String, :required => true
+      property :url, Text
+
+      has n, :entities
+    end
+
+  end
+end
diff --git a/server/lib/deltacloud/drivers/base_driver.rb b/server/lib/deltacloud/drivers/base_driver.rb
index cfb689d..b828d33 100644
--- a/server/lib/deltacloud/drivers/base_driver.rb
+++ b/server/lib/deltacloud/drivers/base_driver.rb
@@ -60,6 +60,10 @@ module Deltacloud
       features.has_key?(collection) and features[collection].include?(feature_name)
     end
 
+    def has_feature?(collection, feature_name)
+      self.class.has_feature?(collection, feature_name)
+    end
+
     def name
       self.class.name.split('::').last.gsub('Driver', '').downcase
     end
diff --git a/server/lib/deltacloud/models/base_model.rb b/server/lib/deltacloud/models/base_model.rb
index 5c19ad5..2553042 100644
--- a/server/lib/deltacloud/models/base_model.rb
+++ b/server/lib/deltacloud/models/base_model.rb
@@ -44,4 +44,8 @@ class BaseModel
     @id
   end
 
+  def to_entity
+    self.class.name.downcase
+  end
+
 end
-- 
1.8.0