You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by Chris Lalancette <cl...@redhat.com> on 2011/08/17 15:57:56 UTC
Blob creation
Hey Marios,
I know this is several months out of date, but I was just doing some
testing on the blob creation stuff and noticing that my libdeltacloud tests
were failing. I traced it down to the fact that the blob_id parameter changed
from param[:blob_id] to param[:blob] when you added the streaming stuff to
blobs.
If I remember correctly, deltacloud 0.3.0 was released with bucket/blob
stuff (even if it wasn't streaming), so I think we'll probably want to
re-instate :blob_id as a parameter to keep backwards compatibility. What do
you think?
--
Chris Lalancette
Re: Blob creation
Posted by Chris Lalancette <cl...@redhat.com>.
On 08/17/11 - 09:57:56AM, Chris Lalancette wrote:
> Hey Marios,
> I know this is several months out of date, but I was just doing some
> testing on the blob creation stuff and noticing that my libdeltacloud tests
> were failing. I traced it down to the fact that the blob_id parameter changed
> from param[:blob_id] to param[:blob] when you added the streaming stuff to
> blobs.
> If I remember correctly, deltacloud 0.3.0 was released with bucket/blob
> stuff (even if it wasn't streaming), so I think we'll probably want to
> re-instate :blob_id as a parameter to keep backwards compatibility. What do
> you think?
Maybe like this patch?
--
Chris Lalancette
Re: Blob creation
Posted by David Lutterkort <lu...@redhat.com>.
On Thu, 2011-08-18 at 11:07 +0300, marios@redhat.com wrote:
> The name change for the parameter was, I can only guess, some attempt to
> maintain consistency (i.e. 'blob' over 'blob_id') though in hindsight
> was not really necessary. Your suggested patch:
>
>
> post "#{Sinatra::UrlForHelper::DEFAULT_URI_PREFIX}/buckets/:bucket" do
> bucket_id = params[:bucket]
> - blob_id = params['blob']
> + blob_id = params['blob'] || params['blob_id']
>
> seems fine to me in that it won't break anything. If it maintains
> compatibility with your stuff then I personally have no objection to
> making this addition. More on PUT vs POST below
Since going from blob_id to blob is API breakage, our docs etc. should
only mention blob_id, and we should remove accepting a blob param
alltogether.
David
Re: Blob creation
Posted by Chris Lalancette <cl...@redhat.com>.
On 08/18/11 - 11:07:57AM, marios@redhat.com wrote:
> Hi Chris, (inline)
>
> On 17/08/11 21:12, David Lutterkort wrote:
> >On Wed, 2011-08-17 at 09:57 -0400, Chris Lalancette wrote:
> >>Hey Marios,
> >> I know this is several months out of date, but I was just doing some
> >>testing on the blob creation stuff and noticing that my libdeltacloud tests
> >>were failing. I traced it down to the fact that the blob_id parameter changed
> >>from param[:blob_id] to param[:blob] when you added the streaming stuff to
> >>blobs.
>
> Yes thats right. Initially we had one operation for creating blobs:
>
> POST /api/buckets/:bucket
>
> and this accepted (amongst others) the 'blob_id' parameter to define
> the name of the blob
>
> Then, in order to implement streaming PUT through deltacloud I added:
>
> PUT /api/buckets/:bucket/:blob
>
> The name change for the parameter was, I can only guess, some
> attempt to maintain consistency (i.e. 'blob' over 'blob_id') though
> in hindsight was not really necessary. Your suggested patch:
>
>
> post "#{Sinatra::UrlForHelper::DEFAULT_URI_PREFIX}/buckets/:bucket" do
> bucket_id = params[:bucket]
> - blob_id = params['blob']
> + blob_id = params['blob'] || params['blob_id']
>
> seems fine to me in that it won't break anything. If it maintains
> compatibility with your stuff then I personally have no objection to
> making this addition. More on PUT vs POST below
OK, thanks. I pushed this change into the repository.
--
Chris Lalancette
Re: Blob creation
Posted by "marios@redhat.com" <ma...@redhat.com>.
Hi Chris, (inline)
On 17/08/11 21:12, David Lutterkort wrote:
> On Wed, 2011-08-17 at 09:57 -0400, Chris Lalancette wrote:
>> Hey Marios,
>> I know this is several months out of date, but I was just doing some
>> testing on the blob creation stuff and noticing that my libdeltacloud tests
>> were failing. I traced it down to the fact that the blob_id parameter changed
>> from param[:blob_id] to param[:blob] when you added the streaming stuff to
>> blobs.
Yes thats right. Initially we had one operation for creating blobs:
POST /api/buckets/:bucket
and this accepted (amongst others) the 'blob_id' parameter to define the
name of the blob
Then, in order to implement streaming PUT through deltacloud I added:
PUT /api/buckets/:bucket/:blob
The name change for the parameter was, I can only guess, some attempt to
maintain consistency (i.e. 'blob' over 'blob_id') though in hindsight
was not really necessary. Your suggested patch:
post "#{Sinatra::UrlForHelper::DEFAULT_URI_PREFIX}/buckets/:bucket" do
bucket_id = params[:bucket]
- blob_id = params['blob']
+ blob_id = params['blob'] || params['blob_id']
seems fine to me in that it won't break anything. If it maintains
compatibility with your stuff then I personally have no objection to
making this addition. More on PUT vs POST below
>
> I think it's another case where the code does somehing special for the
> HTML UI - the official API for creating a new blob is
> PUT /api/buckets/:bucket/:blob; looking at this now, it seems strange
> that we have two different ways to create blobs, and I am wondering if
> we shouldn't drop the PUT, and only use POST for everything.
>
Yes, we have two methods for creating blobs: POST
(http://incubator.apache.org/deltacloud/api#h4_3_8) and PUT
(http://incubator.apache.org/deltacloud/api#h4_3_7).
The POST method is non-streaming:
client ---TEMP_FILE---> deltacloud ---STREAM---> provider
i.e., the client sends the blob to deltacloud, which receives the entire
request and creates a temp_file for the blob data, and then streams this
to the provider.
The PUT operation is streaming:
client ---STREAM---> deltacloud ---STREAM---> provider
i.e., the client sends the blob to deltacloud, which does not wait to
receive the entire request and instead starts streaming the blob data to
the provider as this is received.
Now, in order to create a blob on a given cloud provider service, you
invariably must specify the content_length of the blob. For a PUT
operation, the content_length is exactly as defined by the sending
client in the PUT to deltacloud. Thus, we can take that content_length
and start sending the data to the provider as we are receiving it.
However, for a POST operation, the content_length of the blob is not
what is sent for the client POST operation to deltacloud, due to the
presence of the multipart/form-data boundary, which will vary depending
on the sending client. It became very messy/difficult to try and parse
the boundary and 'guess' the content length of the blob in order to
start streaming, which is why we decided to go with PUT. In fact, the
cloud providers themselves (EC2, rackspace, Azure) use PUT operations to
create blobs (with POST supported as an alternative).
Thus, we have both POST (non streaming, only to support HTML forms and
the web browser interface) and PUT (streaming). If we want to remove one
of those methods then I would definitely vote to remove POST since imho
the streaming functionality for creating blobs is absolutely necessary
for 'real world' use. Forcing deltacloud to buffer all blob objects
before sending them on to the provider is obviously not very useful.
marios
> David
>
>
Re: Blob creation
Posted by David Lutterkort <lu...@redhat.com>.
On Wed, 2011-08-17 at 09:57 -0400, Chris Lalancette wrote:
> Hey Marios,
> I know this is several months out of date, but I was just doing some
> testing on the blob creation stuff and noticing that my libdeltacloud tests
> were failing. I traced it down to the fact that the blob_id parameter changed
> from param[:blob_id] to param[:blob] when you added the streaming stuff to
> blobs.
I think it's another case where the code does somehing special for the
HTML UI - the official API for creating a new blob is
PUT /api/buckets/:bucket/:blob; looking at this now, it seems strange
that we have two different ways to create blobs, and I am wondering if
we shouldn't drop the PUT, and only use POST for everything.
David