You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by ma...@redhat.com on 2011/01/21 20:34:24 UTC
user metadata for blob creation - S3, Cloudfiles, Azure
This adds the ability to put arbitrary user metadata during blob creation for Amazon S3, Rackspace Cloudfiles and Microsoft Azure.
As previously mentioned, this doesn't include the ability to update/delete individual metadata (but planned for later - this patch is to be committed as a baseline). The main issue is with S3. Whilst Cloudfiles and Azure provide api calls for updating and deleting metadata after a blob is created, for S3 it is only possible to do so when creating the blob. Apparently, S3 clients such as cloudberry explorer which allow you to update metadata actually *make a copy of the object with the new metadata* (!) https://forums.aws.amazon.com/thread.jspa?messageID=172472𪆸
Another issue which is not yet settled and for which I'd be grateful for feedback is in server.rb I am using params to get the key-value pairs for the metadata (submitted by a html form). An alternative, and the way the providers do it is with custom request headers (X-AMZ-META-KEY = VALUE for S3, X-OBJECT-META-KEY = VALUE for rackspace etc). This needs some more thought,
marios
Re: user metadata for blob creation - S3, Cloudfiles, Azure
Posted by David Lutterkort <lu...@redhat.com>.
On Fri, 2011-01-21 at 21:34 +0200, marios@redhat.com wrote:
> Another issue which is not yet settled and for which I'd be grateful
> for feedback is in server.rb I am using params to get the key-value
> pairs for the metadata (submitted by a html form). An alternative, and
> the way the providers do it is with custom request headers
> (X-AMZ-META-KEY = VALUE for S3, X-OBJECT-META-KEY = VALUE for
> rackspace etc). This needs some more thought,
I don't have any strong feelings either way, but if the providers do it,
might be most natural to do it that way in Deltacloud too, and accept
X-DC-META-KEY headers.
Of course, that will make it next to impossible to set metadata from the
browser, but the HTML interface is really for exploration, and not for
serious uses of Deltacloud.
David
Re: [PATCH] Adds ability to specify arbitrary user metadata (KEY-VALUE)
for S3, Cloudfiles and Azure, during blob creation.
Posted by "marios@redhat.com" <ma...@redhat.com>.
On 22/01/11 02:14, David Lutterkort wrote:
> Some comments below:
>> + def replace_hash_keys(hash, pattern)
>
> Nit pick: what you call 'pattern' is really the replacement.
>
> The Ruby way to do something like this is to define it directly in the
> Hash class, i.e. doing something like
>
> class Hash
> def replace_keys(pattern)
> ...
> end
> end
>
> though it might be clearer to have a method Hash.gsub_keys(pattern,
> replacement)
>
cool :) thanks for your comments, I'll rework the patch and send out
again (including the x-deltacloud-meta headers tweak for passing in the
metadata values - i'm thinking we can have both mechanisms [params and
headers] i.e. just add the checking for x-deltacloud-meta headers to the
code.. i'll see how this works in practice),
marios
>> end
>>
>> end
>> diff --git a/server/lib/deltacloud/drivers/azure/azure_driver.rb b/server/lib/deltacloud/drivers/azure/azure_driver.rb
>> index abbd353..20c6295 100644
>> --- a/server/lib/deltacloud/drivers/azure/azure_driver.rb
>> +++ b/server/lib/deltacloud/drivers/azure/azure_driver.rb
>> @@ -94,16 +94,22 @@ class AzureDriver< Deltacloud::BaseDriver
>> #--
>> # Create Blob
>> #--
>> +require 'ruby-debug'
>> def create_blob(credentials, bucket_id, blob_id, blob_data, opts=nil)
>
> Default value for opts should be {}, so that replace_keys doesn't barf
> on a nil hash
>
> Other than that, and the issue of how to pass in metadata k/v pairs,
> looks good.
>
> David
>
>
Re: [PATCH] Adds ability to specify arbitrary user metadata
(KEY-VALUE) for S3, Cloudfiles and Azure, during blob creation.
Posted by David Lutterkort <lu...@redhat.com>.
Some comments below:
On Fri, 2011-01-21 at 21:34 +0200, marios@redhat.com wrote:
> diff --git a/server/lib/deltacloud/base_driver/base_driver.rb b/server/lib/deltacloud/base_driver/base_driver.rb
> index bf06e56..5bae2f1 100644
> --- a/server/lib/deltacloud/base_driver/base_driver.rb
> +++ b/server/lib/deltacloud/base_driver/base_driver.rb
> @@ -252,6 +252,19 @@ module Deltacloud
> end
> end
>
> + #used by blobstore - server.rb prefixes all user metadata keys with "META-"
> + #but each backend blobstore defines their own meta header - e.g. amazon is "x-amz-meta-"
> + def replace_hash_keys(hash, pattern)
Nit pick: what you call 'pattern' is really the replacement.
> + hash.each do |k, v|
> + if k.match(/^META-/)
> + new_k = k.gsub(/^META-/, pattern)
> + hash[new_k] = v
> + hash.delete(k)
Is it really safe in Ruby to modify a hash while iterating over it ?
> + end #if
> + end#opts.each
> + return hash
> + end
> +
The Ruby way to do something like this is to define it directly in the
Hash class, i.e. doing something like
class Hash
def replace_keys(pattern)
...
end
end
though it might be clearer to have a method Hash.gsub_keys(pattern,
replacement)
> end
>
> end
> diff --git a/server/lib/deltacloud/drivers/azure/azure_driver.rb b/server/lib/deltacloud/drivers/azure/azure_driver.rb
> index abbd353..20c6295 100644
> --- a/server/lib/deltacloud/drivers/azure/azure_driver.rb
> +++ b/server/lib/deltacloud/drivers/azure/azure_driver.rb
> @@ -94,16 +94,22 @@ class AzureDriver < Deltacloud::BaseDriver
> #--
> # Create Blob
> #--
> +require 'ruby-debug'
> def create_blob(credentials, bucket_id, blob_id, blob_data, opts=nil)
Default value for opts should be {}, so that replace_keys doesn't barf
on a nil hash
Other than that, and the issue of how to pass in metadata k/v pairs,
looks good.
David
[PATCH] Adds ability to specify arbitrary user metadata (KEY-VALUE) for S3, Cloudfiles and Azure, during blob creation.
Posted by ma...@redhat.com.
From: marios <ma...@redhat.com>
---
server/lib/deltacloud/base_driver/base_driver.rb | 13 +++++++
.../lib/deltacloud/drivers/azure/azure_driver.rb | 17 +++++++---
server/lib/deltacloud/drivers/ec2/ec2_driver.rb | 13 +++++--
.../drivers/rackspace/rackspace_driver.rb | 23 +++++++++----
server/lib/deltacloud/models/blob.rb | 1 +
server/public/javascripts/application.js | 35 ++++++++++++++++++++
server/server.rb | 10 +++++-
server/views/blobs/new.html.haml | 18 +++++++++-
server/views/blobs/show.html.haml | 3 ++
server/views/blobs/show.xml.haml | 4 ++-
10 files changed, 118 insertions(+), 19 deletions(-)
diff --git a/server/lib/deltacloud/base_driver/base_driver.rb b/server/lib/deltacloud/base_driver/base_driver.rb
index bf06e56..5bae2f1 100644
--- a/server/lib/deltacloud/base_driver/base_driver.rb
+++ b/server/lib/deltacloud/base_driver/base_driver.rb
@@ -252,6 +252,19 @@ module Deltacloud
end
end
+ #used by blobstore - server.rb prefixes all user metadata keys with "META-"
+ #but each backend blobstore defines their own meta header - e.g. amazon is "x-amz-meta-"
+ def replace_hash_keys(hash, pattern)
+ hash.each do |k, v|
+ if k.match(/^META-/)
+ new_k = k.gsub(/^META-/, pattern)
+ hash[new_k] = v
+ hash.delete(k)
+ end #if
+ end#opts.each
+ return hash
+ end
+
end
end
diff --git a/server/lib/deltacloud/drivers/azure/azure_driver.rb b/server/lib/deltacloud/drivers/azure/azure_driver.rb
index abbd353..20c6295 100644
--- a/server/lib/deltacloud/drivers/azure/azure_driver.rb
+++ b/server/lib/deltacloud/drivers/azure/azure_driver.rb
@@ -94,16 +94,22 @@ class AzureDriver < Deltacloud::BaseDriver
#--
# Create Blob
#--
+require 'ruby-debug'
def create_blob(credentials, bucket_id, blob_id, blob_data, opts=nil)
azure_connect(credentials)
- #get a handle to the bucket in order to put there
- the_bucket = WAZ::Blobs::Container.find(bucket_id)
- the_bucket.store(blob_id, blob_data[:tempfile], blob_data[:type])
+ #insert azure-specific header for user metadata ... x-ms-meta-kEY = VALUE
+ opts = replace_hash_keys(opts, "x-ms-meta-")
+ safely do
+ #get a handle to the bucket in order to put there
+ the_bucket = WAZ::Blobs::Container.find(bucket_id)
+ the_bucket.store(blob_id, blob_data[:tempfile], blob_data[:type], opts)
+ end
Blob.new( { :id => blob_id,
:bucket => bucket_id,
:content_lengh => blob_data[:tempfile].length,
:content_type => blob_data[:type],
- :last_modified => ''
+ :last_modified => '',
+ :user_metadata => opts.select{|k,v| k.match(/^x-ms-meta-/)}
} )
end
@@ -146,7 +152,8 @@ class AzureDriver < Deltacloud::BaseDriver
:bucket => bucket,
:content_length => waz_blob.metadata[:content_length],
:content_type => waz_blob.metadata[:content_type],
- :last_modified => waz_blob.metadata[:last_modified]
+ :last_modified => waz_blob.metadata[:last_modified],
+ :user_metadata => waz_blob.metadata.select{|k,v| k.to_s.match(/^x_ms_meta_/)}
})
end
diff --git a/server/lib/deltacloud/drivers/ec2/ec2_driver.rb b/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
index 7a4b394..db0e1a1 100644
--- a/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
+++ b/server/lib/deltacloud/drivers/ec2/ec2_driver.rb
@@ -361,18 +361,22 @@ module Deltacloud
res = nil
# File stream needs to be reopened in binary mode for whatever reason
file = File::open(data[:tempfile].path, 'rb')
+ #insert ec2-specific header for user metadata ... x-amz-meta-KEY = VALUE
+ opts = replace_hash_keys(opts, "x-amz-meta-")
+ opts["Content-Type"] = data[:type]
safely do
res = s3_client.interface.put(bucket_id,
blob_id,
file,
- {"Content-Type" => data[:type]})
+ opts)
end
#create a new Blob object and return that
Blob.new( { :id => blob_id,
:bucket => bucket_id,
:content_length => data[:tempfile].length,
:content_type => data[:type],
- :last_modified => ''
+ :last_modified => '',
+ :user_metadata => opts.select{|k,v| k.match(/^x-amz-meta-/)}
}
)
end
@@ -503,7 +507,7 @@ module Deltacloud
'us-east-1' => 'elasticloadbalancing.us-east-1.amazonaws.com',
'us-west-1' => 'elasticloadbalancing.us-west-1.amazonaws.com'
},
-
+
's3' => {
'ap-southeast-1' => 's3.amazonaws.com',
'eu-west-1' => 's3.amazonaws.com',
@@ -568,7 +572,8 @@ module Deltacloud
:bucket => s3_object.bucket.name.to_s,
:content_length => s3_object.headers['content-length'],
:content_type => s3_object.headers['content-type'],
- :last_modified => s3_object.last_modified
+ :last_modified => s3_object.last_modified,
+ :user_metadata => s3_object.meta_headers
)
end
diff --git a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
index e40fb9c..9a45d4f 100644
--- a/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
+++ b/server/lib/deltacloud/drivers/rackspace/rackspace_driver.rb
@@ -233,16 +233,24 @@ class RackspaceDriver < Deltacloud::BaseDriver
#--
def create_blob(credentials, bucket_id, blob_id, blob_data, opts=nil)
cf = cloudfiles_client(credentials)
- #must first create the object using cloudfiles_client.create_object
- #then can write using object.write(data)
- object = cf.container(bucket_id).create_object(blob_id)
- #blob_data is a construct with data in .tempfile and content-type in {:type}
- res = object.write(blob_data[:tempfile], {'Content-Type' => blob_data[:type]})
+ user_meta = opts.select{|k,v| k.match(/^META-/)}
+ #insert ec2-specific header for user metadata ... X-Object-Meta-KEY = VALUE
+ opts = replace_hash_keys(opts, "X-Object-Meta-")
+ opts['Content-Type'] = blob_data[:type]
+ object = nil
+ safely do
+ #must first create the object using cloudfiles_client.create_object
+ #then can write using object.write(data)
+ object = cf.container(bucket_id).create_object(blob_id)
+ #blob_data is a construct with data in .tempfile and content-type in {:type}
+ res = object.write(blob_data[:tempfile], opts)
+ end
Blob.new( { :id => object.name,
:bucket => object.container.name,
:content_length => blob_data[:tempfile].length,
:content_type => blob_data[:type],
- :last_modified => ''
+ :last_modified => '',
+ :user_metadata => opts.select{|k,v| k.match(/^X-Object-Meta-/)}
}
)
end
@@ -292,7 +300,8 @@ private
:bucket => cf_object.container.name,
:content_length => cf_object.bytes,
:content_type => cf_object.content_type,
- :last_modified => cf_object.last_modified
+ :last_modified => cf_object.last_modified,
+ :user_metadata => cf_object.metadata
})
end
diff --git a/server/lib/deltacloud/models/blob.rb b/server/lib/deltacloud/models/blob.rb
index dfa67fe..0bf473b 100644
--- a/server/lib/deltacloud/models/blob.rb
+++ b/server/lib/deltacloud/models/blob.rb
@@ -24,5 +24,6 @@ class Blob < BaseModel
attr_accessor :content_type
attr_accessor :last_modified
attr_accessor :content
+ attr_accessor :user_metadata
end
diff --git a/server/public/javascripts/application.js b/server/public/javascripts/application.js
index 5ee0f7f..95c9bc2 100644
--- a/server/public/javascripts/application.js
+++ b/server/public/javascripts/application.js
@@ -16,3 +16,38 @@ $(document).ready(function() {
}
})
+
+function more_fields()
+{
+ //increment the hidden input that captures how many meta_data are passed
+ var meta_params = document.getElementsByName('meta_params')
+ current_number_params = eval(meta_params[0].value)+1
+ meta_params[0].value = current_number_params
+ var new_meta = document.getElementById('metadata_holder').cloneNode(true);
+ new_meta.id = 'metadata_holder' + current_number_params;
+ new_meta.style.display = 'block';
+ var nodes = new_meta.childNodes;
+ for (var i=0;i < nodes.length;i++) {
+ var theName = nodes[i].name;
+ if (theName)
+ nodes[i].name = theName + current_number_params;
+ }
+ var insertHere = document.getElementById('metadata_holder');
+ insertHere.parentNode.insertBefore(new_meta,insertHere);
+}
+
+function less_fields()
+{
+ var meta_params = document.getElementsByName('meta_params')
+ current_val = eval(meta_params[0].value)
+ if (current_val == 0)
+ {
+ return;
+ }
+ else
+ {
+ var theDiv = document.getElementById('metadata_holder'+current_val)
+ theDiv.parentNode.removeChild(theDiv)
+ meta_params[0].value = eval(current_val)-1
+ }
+}
diff --git a/server/server.rb b/server/server.rb
index 8c3b72c..418a497 100644
--- a/server/server.rb
+++ b/server/server.rb
@@ -589,7 +589,15 @@ post '/api/buckets/:bucket' do
bucket_id = params[:bucket]
blob_id = params['blob_id']
blob_data = params['blob_data']
- @blob = driver.create_blob(credentials, bucket_id, blob_id, blob_data )
+ max = params[:meta_params].to_i
+ user_meta = {}
+ (1..max).each do |i|
+ key = params[:"meta_name#{i}"]
+ key = "META-#{key}"
+ value = params[:"meta_value#{i}"]
+ user_meta[key] = value
+ end
+ @blob = driver.create_blob(credentials, bucket_id, blob_id, blob_data, user_meta)
respond_to do |format|
format.html { haml :"blobs/show"}
format.xml { haml :"blobs/show" }
diff --git a/server/views/blobs/new.html.haml b/server/views/blobs/new.html.haml
index feb94e5..a9817d5 100644
--- a/server/views/blobs/new.html.haml
+++ b/server/views/blobs/new.html.haml
@@ -4,7 +4,23 @@
%label
Blob Name:
%input{ :name => 'blob_id', :size => 512}/
+ %label
Blob Data:
- %input{ :type => "file", :name => 'blob_data', :size => 50}/
%br
+ %input{ :type => "file", :name => 'blob_data', :size => 50}/
+ %br
+ %br
+ %input{ :type => "hidden", :name => "meta_params", :value => "0"}
+ %a{ :href => "javascript:;", :onclick => "more_fields();"} Add Metadata
+ %div{ :id => "metadata_holder", :style => "display: none;"}
+ %label
+ Metadata Name:
+ %input{ :type => "text", :name => "meta_name", :value => ""}/
+ %label
+ Metadata Value:
+ %input{ :type => "text", :name => "meta_value", :value => ""}/
+ %br
+ %a{ :href => "javascript:;", :onclick => "less_fields();"} Less Metadata
+ %br
+ %br
%input{ :type => :submit, :name => "commit", :value => "create"}/
diff --git a/server/views/blobs/show.html.haml b/server/views/blobs/show.html.haml
index e5c2cee..29025c0 100644
--- a/server/views/blobs/show.html.haml
+++ b/server/views/blobs/show.html.haml
@@ -19,6 +19,9 @@
%dt Content
%dd
=link_to 'Blob content', bucket_url(@blob.bucket) + '/' + @blob.id + '/content'
+ %dt User_Metadata
+ %dd
+ = @blob.user_metadata.inspect
%dt Delete this Blob
%dd
%form{ :action => bucket_url(@blob.bucket) + '/' + @blob.id , :method => :post }
diff --git a/server/views/blobs/show.xml.haml b/server/views/blobs/show.xml.haml
index bc499ca..1b9ea55 100644
--- a/server/views/blobs/show.xml.haml
+++ b/server/views/blobs/show.xml.haml
@@ -1,7 +1,9 @@
!!! XML
%blob{:href => bucket_url(@blob.bucket) + '/' + @blob.id, :id => @blob.id}
- - @blob.attributes.select{ |attr| attr!=:id}.each do |attribute|
+ - @blob.attributes.select{ |attr| (attr!=:id && attr!=:user_metadata) }.each do |attribute|
- unless attribute == :content
- haml_tag(attribute, :<) do
- haml_concat @blob.send(attribute)
+ - @blob.user_metadata.each do |k, v|
+ %user_metadata{:name => k, :value => v}
%content{:href => bucket_url(@blob.bucket) + '/' + @blob.id + '/content'}
--
1.7.3.4
Re: user metadata for blob creation - S3, Cloudfiles, Azure
Posted by "marios@redhat.com" <ma...@redhat.com>.
On 21/01/11 21:34, marios@redhat.com wrote:
> This adds the ability to put arbitrary user metadata during blob creation for Amazon S3, Rackspace Cloudfiles and Microsoft Azure.
>
> As previously mentioned, this doesn't include the ability to update/delete individual metadata (but planned for later - this patch is to be committed as a baseline). The main issue is with S3. Whilst Cloudfiles and Azure provide api calls for updating and deleting metadata after a blob is created, for S3 it is only possible to do so when creating the blob. Apparently, S3 clients such as cloudberry explorer which allow you to update metadata actually *make a copy of the object with the new metadata* (!) https://forums.aws.amazon.com/thread.jspa?messageID=172472𪆸
>
> Another issue which is not yet settled and for which I'd be grateful for feedback is in server.rb I am using params to get the key-value pairs for the metadata (submitted by a html form). An alternative, and the way the providers do it is with custom request headers (X-AMZ-META-KEY = VALUE for S3, X-OBJECT-META-KEY = VALUE for rackspace etc). This needs some more thought,
>
> marios
This patchset has been succeeded by new version which include the
comments here - 25th January 'user metadata for blob creation [v.2.0] -
S3, Cloudfiles, Azure'
marios