You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by ma...@redhat.com on 2011/02/02 21:05:36 UTC
Get and Set user metadata for blobs
This patch adds get and set (user defined) blob metadata where supported - MSFT Azure and Rackspace Cloudfiles:
Get blob metadata: HEAD /api/buckets/myBucket/myBlob
First of all, is HEAD the right approach here? We could for instance do something like GET /api/buckets/myBucket/myBlob/metadata. For HEAD and as per rfc 2616 (section 9.5) we do NOT return a message-body - instead the blob metadata is reported in the response headers (X-Deltacloud-Blobmeta-KEY:VALUE)
Set blob metadata: POST /api/buckets/myBucket/myBlob
Again here I do not return a response body but rather report the new blob metadata in the response headers. This is another thing up for discussion; we could for instance make a seperate call to request the (now updated) blob and return that in the response body?
Note: if you intend to test the Azure driver
There was an ommission in the azure gem (latest 1.0.3) which caused problems with setting metadata. I've made the fix and issued a pull request to the author ( https://github.com/johnnyhalife/waz-storage/pull/7 ). I don't know how long it will take for the change to be pulled in so if you'd like to use this functionality beforehand then you can use my fork at: git://github.com/marios/waz-storage.git (git clone this).
marios
Re: [PATCH] Adds get and set (user-defined) blob metadata for azure
and cloudfiles drivers
Posted by "marios@redhat.com" <ma...@redhat.com>.
On 07/02/11 17:45, Michal Fojtik wrote:
> On 02/02/11 22:05 +0200, marios@redhat.com wrote:
>
> ACK. Code looks good, just small inline comment about returning some
> content / status code from HEAD and POST blocks.
>
>> +#get blob metadata
>> +head '/api/buckets/:bucket/:blob' do
>> + @blob_id = params[:blob]
>> + @blob_metadata = driver.blob_metadata(credentials, {:id =>
>> params[:blob], 'bucket' => params[:bucket]})
>> + if @blob_metadata
>> + @blob_metadata.each do |k,v|
>> + headers["X-Deltacloud-Blobmeta-#{k}"] = v
>> + end
>> + else
>> + report_error(404, 'not_found')
>> + end
>
> Please return something here, otherwise the 'each' return value will be
> sent ;-) I'm used to return '[200, '']' on HEAD calls.
>
I am returning something - I'm setting the response headers to include
HTTP-X-Deltacloud-Blobmeta-KEY:Value for all blob metadata pairs.
<Copy_Paste>:
curl -I --user 'username:password'
http://localhost:3001/api/buckets/mariosshinymsftbucket/anamazingimportantfile?format=xml
HTTP/1.1 200 OK
X-Runtime: 2.722210
X-Deltacloud-Blobmeta-mynewkey: MYNEWVALUE
Content-Type: text/html
Connection: close
Server: thin 1.2.7 codename No Hup
</Copy_Paste>
marios
Re: [PATCH] Adds get and set (user-defined) blob metadata for
azure and cloudfiles drivers
Posted by Michal Fojtik <mf...@redhat.com>.
On 02/02/11 22:05 +0200, marios@redhat.com wrote:
ACK. Code looks good, just small inline comment about returning some
content / status code from HEAD and POST blocks.
-- Michal
>From: marios <ma...@redhat.com>
>
>---
> .../lib/deltacloud/drivers/azure/azure_driver.rb | 28 +++++++++++++++++++
> .../drivers/rackspace/rackspace_driver.rb | 29 +++++++++++++++++--
> server/lib/deltacloud/helpers/blob_stream.rb | 7 ++---
> server/server.rb | 27 ++++++++++++++++++
> 4 files changed, 84 insertions(+), 7 deletions(-)
>
>diff --git a/server/lib/deltacloud/drivers/azure/azure_driver.rb b/server/lib/deltacloud/drivers/azure/azure_driver.rb
>index f17e11e..704fbc5 100644
>--- a/server/lib/deltacloud/drivers/azure/azure_driver.rb
>+++ b/server/lib/deltacloud/drivers/azure/azure_driver.rb
>@@ -128,6 +128,34 @@ class AzureDriver < Deltacloud::BaseDriver
> the_blob.destroy!
> end
>
>+#-
>+# Blob Metadada
>+#-
>+ def blob_metadata(credentials, opts = {})
>+ azure_connect(credentials)
>+ all_meta = {}
>+ safely do
>+ all_meta = WAZ::Blobs::Container.find(opts['bucket'])[opts[:id]].metadata
>+ end
>+ user_meta = {}
>+ all_meta.inject({}){|result_hash, (k,v)| user_meta[k]=v if k.to_s.match(/x_ms_meta/i)}
>+ user_meta.gsub_keys('x_ms_meta_','')
>+ end
>+
>+#-
>+# Update Blob Metadata
>+#-
>+ def update_blob_metadata(credentials, opts={})
>+ azure_connect(credentials)
>+ meta_hash = opts['meta_hash']
>+ meta_hash.gsub_keys("HTTP_X_Deltacloud_Blobmeta_", "x-ms-meta-")
>+ safely do
>+ the_blob = WAZ::Blobs::Container.find(opts['bucket'])[opts[:id]]
>+ the_blob.put_metadata!(meta_hash)
>+ end
>+ end
>+
>+
> private
>
> def azure_connect(credentials)
>diff --git a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
>index eba8f8f..c09e36a 100644
>--- a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
>+++ b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
>@@ -147,9 +147,6 @@ class RackspaceDriver < Deltacloud::BaseDriver
> true
> end
>
>-
>-
>-
> define_instance_states do
> start.to( :pending ) .on( :create )
>
>@@ -262,6 +259,32 @@ class RackspaceDriver < Deltacloud::BaseDriver
> cf.container(bucket_id).delete_object(blob_id)
> end
>
>+#-
>+# Blob Metadada
>+#-
>+ def blob_metadata(credentials, opts = {})
>+ cf = cloudfiles_client(credentials)
>+ meta = {}
>+ safely do
>+ meta = cf.container(opts['bucket']).object(opts[:id]).metadata
>+ end
>+ meta
>+ end
>+
>+#-
>+# Update Blob Metahash
>+#-
>+ def update_blob_metadata(credentials, opts={})
>+ cf = cloudfiles_client(credentials)
>+ meta_hash = opts['meta_hash']
>+ #the set_metadata method actually places the 'X-Object-Meta-' prefix for us:
>+ meta_hash.gsub_keys('HTTP_X_Deltacloud_Blobmeta_', '')
>+ safely do
>+ blob = cf.container(opts['bucket']).object(opts[:id])
>+ blob.set_metadata(meta_hash)
>+ end #safely
>+ end
>+
> private
>
> def convert_srv_to_instance(srv)
>diff --git a/server/lib/deltacloud/helpers/blob_stream.rb b/server/lib/deltacloud/helpers/blob_stream.rb
>index 985bbc6..a0dfd42 100644
>--- a/server/lib/deltacloud/helpers/blob_stream.rb
>+++ b/server/lib/deltacloud/helpers/blob_stream.rb
>@@ -66,12 +66,11 @@ end
> class Hash
>
> def gsub_keys(pattern, replacement)
>- pattern.upcase!
>- replacement.upcase!
>+ rgx_pattern = Regexp.compile(pattern, true)
> remove = []
> self.each_key do |key|
>- if key.to_s.match(pattern)
>- new_key = key.to_s.gsub(pattern, replacement)
>+ if key.to_s.match(rgx_pattern)
>+ new_key = key.to_s.gsub(rgx_pattern, replacement)
> self[new_key] = self[key]
> remove << key
> end #key.match
>diff --git a/server/server.rb b/server/server.rb
>index a801bea..3f092e4 100644
>--- a/server/server.rb
>+++ b/server/server.rb
>@@ -655,6 +655,33 @@ delete '/api/buckets/:bucket/:blob' do
> redirect(bucket_url(bucket_id))
> end
>
>+#get blob metadata
>+head '/api/buckets/:bucket/:blob' do
>+ @blob_id = params[:blob]
>+ @blob_metadata = driver.blob_metadata(credentials, {:id => params[:blob], 'bucket' => params[:bucket]})
>+ if @blob_metadata
>+ @blob_metadata.each do |k,v|
>+ headers["X-Deltacloud-Blobmeta-#{k}"] = v
>+ end
>+ else
>+ report_error(404, 'not_found')
>+ end
Please return something here, otherwise the 'each' return value will be
sent ;-) I'm used to return '[200, '']' on HEAD calls.
>+end
>+
>+#update blob metadata
>+post '/api/buckets/:bucket/:blob' do
>+ meta_hash = {}
>+ request.env.inject({}){|current, (k,v)| meta_hash[k] = v if k.match(/^HTTP[-_]X[-_]Deltacloud[-_]Blobmeta[-_]/i)}
>+ success = driver.update_blob_metadata(credentials, {'bucket'=>params[:bucket], :id =>params[:blob], 'meta_hash' => meta_hash})
>+ if(success)
>+ meta_hash.each do |k,v|
>+ headers["X-Deltacloud-Blobmeta-#{k}"] = v
>+ end
>+ else
>+ report_error(404, 'not_found') #FIXME is this the right error code?
>+ end
>+end
Same stuff as above.
>+
> #Get a particular blob's particulars (not actual blob data)
> get '/api/buckets/:bucket/:blob' do
> @blob = driver.blob(credentials, { :id => params[:blob], 'bucket' => params[:bucket]})
>--
>1.7.3.4
>
--
--------------------------------------------------------
Michal Fojtik, mfojtik@redhat.com
Deltacloud API: http://deltacloud.org
--------------------------------------------------------
[PATCH] Adds get and set (user-defined) blob metadata for azure and cloudfiles drivers
Posted by ma...@redhat.com.
From: marios <ma...@redhat.com>
---
.../lib/deltacloud/drivers/azure/azure_driver.rb | 28 +++++++++++++++++++
.../drivers/rackspace/rackspace_driver.rb | 29 +++++++++++++++++--
server/lib/deltacloud/helpers/blob_stream.rb | 7 ++---
server/server.rb | 27 ++++++++++++++++++
4 files changed, 84 insertions(+), 7 deletions(-)
diff --git a/server/lib/deltacloud/drivers/azure/azure_driver.rb b/server/lib/deltacloud/drivers/azure/azure_driver.rb
index f17e11e..704fbc5 100644
--- a/server/lib/deltacloud/drivers/azure/azure_driver.rb
+++ b/server/lib/deltacloud/drivers/azure/azure_driver.rb
@@ -128,6 +128,34 @@ class AzureDriver < Deltacloud::BaseDriver
the_blob.destroy!
end
+#-
+# Blob Metadada
+#-
+ def blob_metadata(credentials, opts = {})
+ azure_connect(credentials)
+ all_meta = {}
+ safely do
+ all_meta = WAZ::Blobs::Container.find(opts['bucket'])[opts[:id]].metadata
+ end
+ user_meta = {}
+ all_meta.inject({}){|result_hash, (k,v)| user_meta[k]=v if k.to_s.match(/x_ms_meta/i)}
+ user_meta.gsub_keys('x_ms_meta_','')
+ end
+
+#-
+# Update Blob Metadata
+#-
+ def update_blob_metadata(credentials, opts={})
+ azure_connect(credentials)
+ meta_hash = opts['meta_hash']
+ meta_hash.gsub_keys("HTTP_X_Deltacloud_Blobmeta_", "x-ms-meta-")
+ safely do
+ the_blob = WAZ::Blobs::Container.find(opts['bucket'])[opts[:id]]
+ the_blob.put_metadata!(meta_hash)
+ end
+ end
+
+
private
def azure_connect(credentials)
diff --git a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
index eba8f8f..c09e36a 100644
--- a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
+++ b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
@@ -147,9 +147,6 @@ class RackspaceDriver < Deltacloud::BaseDriver
true
end
-
-
-
define_instance_states do
start.to( :pending ) .on( :create )
@@ -262,6 +259,32 @@ class RackspaceDriver < Deltacloud::BaseDriver
cf.container(bucket_id).delete_object(blob_id)
end
+#-
+# Blob Metadada
+#-
+ def blob_metadata(credentials, opts = {})
+ cf = cloudfiles_client(credentials)
+ meta = {}
+ safely do
+ meta = cf.container(opts['bucket']).object(opts[:id]).metadata
+ end
+ meta
+ end
+
+#-
+# Update Blob Metahash
+#-
+ def update_blob_metadata(credentials, opts={})
+ cf = cloudfiles_client(credentials)
+ meta_hash = opts['meta_hash']
+ #the set_metadata method actually places the 'X-Object-Meta-' prefix for us:
+ meta_hash.gsub_keys('HTTP_X_Deltacloud_Blobmeta_', '')
+ safely do
+ blob = cf.container(opts['bucket']).object(opts[:id])
+ blob.set_metadata(meta_hash)
+ end #safely
+ end
+
private
def convert_srv_to_instance(srv)
diff --git a/server/lib/deltacloud/helpers/blob_stream.rb b/server/lib/deltacloud/helpers/blob_stream.rb
index 985bbc6..a0dfd42 100644
--- a/server/lib/deltacloud/helpers/blob_stream.rb
+++ b/server/lib/deltacloud/helpers/blob_stream.rb
@@ -66,12 +66,11 @@ end
class Hash
def gsub_keys(pattern, replacement)
- pattern.upcase!
- replacement.upcase!
+ rgx_pattern = Regexp.compile(pattern, true)
remove = []
self.each_key do |key|
- if key.to_s.match(pattern)
- new_key = key.to_s.gsub(pattern, replacement)
+ if key.to_s.match(rgx_pattern)
+ new_key = key.to_s.gsub(rgx_pattern, replacement)
self[new_key] = self[key]
remove << key
end #key.match
diff --git a/server/server.rb b/server/server.rb
index a801bea..3f092e4 100644
--- a/server/server.rb
+++ b/server/server.rb
@@ -655,6 +655,33 @@ delete '/api/buckets/:bucket/:blob' do
redirect(bucket_url(bucket_id))
end
+#get blob metadata
+head '/api/buckets/:bucket/:blob' do
+ @blob_id = params[:blob]
+ @blob_metadata = driver.blob_metadata(credentials, {:id => params[:blob], 'bucket' => params[:bucket]})
+ if @blob_metadata
+ @blob_metadata.each do |k,v|
+ headers["X-Deltacloud-Blobmeta-#{k}"] = v
+ end
+ else
+ report_error(404, 'not_found')
+ end
+end
+
+#update blob metadata
+post '/api/buckets/:bucket/:blob' do
+ meta_hash = {}
+ request.env.inject({}){|current, (k,v)| meta_hash[k] = v if k.match(/^HTTP[-_]X[-_]Deltacloud[-_]Blobmeta[-_]/i)}
+ success = driver.update_blob_metadata(credentials, {'bucket'=>params[:bucket], :id =>params[:blob], 'meta_hash' => meta_hash})
+ if(success)
+ meta_hash.each do |k,v|
+ headers["X-Deltacloud-Blobmeta-#{k}"] = v
+ end
+ else
+ report_error(404, 'not_found') #FIXME is this the right error code?
+ end
+end
+
#Get a particular blob's particulars (not actual blob data)
get '/api/buckets/:bucket/:blob' do
@blob = driver.blob(credentials, { :id => params[:blob], 'bucket' => params[:bucket]})
--
1.7.3.4
Re: Get and Set user metadata for blobs
Posted by "marios@redhat.com" <ma...@redhat.com>.
On 02/02/11 21:05, marios@redhat.com wrote:
> Note: if you intend to test the Azure driver
> There was an ommission in the azure gem (latest 1.0.3) which caused problems with setting metadata. I've made the fix and issued a pull request to the author ( https://github.com/johnnyhalife/waz-storage/pull/7 ). I don't know how long it will take for the change to be pulled in so if you'd like to use this functionality beforehand then you can use my fork at: git://github.com/marios/waz-storage.git (git clone this).
These changes have now been pulled in by the author (details at
https://github.com/johnnyhalife/waz-storage/pull/7). I've tested the
newest version (1.0.4) today and seems fine - if you are working with
azure driver please re-install for the latest waz-storage gem
marios