You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Jessica Wang <Je...@citrix.com> on 2013/07/01 20:54:30 UTC

RE: Query String Request Authentication(QSRA) support by S3 providers

Thomas,

I checked the data you provided.

The reason that the 2 templates("MyTiny", "AnotherTiny") have no download button is because they are not ready 
(i.e. their "isready" property is false).

Download button is only available when "isready" property is true.

Jessica

-----Original Message-----
From: Thomas O'Dowd [mailto:tpodowd@cloudian.com] 
Sent: Thursday, June 27, 2013 8:04 PM
To: Min Chen
Cc: dev@cloudstack.apache.org; Jessica Wang
Subject: Re: Query String Request Authentication(QSRA) support by S3 providers

Hi Min/Jessica,

I attached an image to that issue to show what what my browser is
showing.

    https://issues.apache.org/jira/browse/CLOUDSTACK-3220 

Tom.

On Fri, 2013-06-28 at 09:45 +0900, Thomas O'Dowd wrote:
> Hi Min,
> 
> Yes. I'll try it again today to check again but when I added Amazon S3
> as the S3 secondary storage and uploaded a template, I was not shown the
> "download template" link. However - for Cloudian S3, I am shown it so
> I'm wondering why.
> 
> Tom.
> 
> On Fri, 2013-06-28 at 00:26 +0000, Min Chen wrote:
> > Hi Tom,
> > 
> > 	Are you saying that you cannot see a Download Template button from UI
> > when Amazon S3 is added as secondary storage? I only tested with RiakCS
> > and Cloudian, so didn't see this issue. But I am CC Jessica her to confirm
> > what special handling is done in UI to enable/disable a button from UI.
> > 
> > 	Thanks
> > 	-min
> > 
> > On 6/27/13 5:23 PM, "Thomas O'Dowd" <tp...@cloudian.com> wrote:
> > 
> > >Hi Min,
> > >
> > >Can you check this bug? I'm trying to test this feature for Amazon but
> > >having no luck getting the Download template link/button to appear.
> > >
> > >https://issues.apache.org/jira/browse/CLOUDSTACK-3220
> > >
> > >Thanks,
> > >
> > >Tom.
> > >
> > >On Fri, 2013-06-21 at 17:21 +0000, Min Chen wrote:
> > >> John,
> > >> 
> > >> 	For S3, the api call createEntityExtractUrl is done on management
> > >>server
> > >> side; while for NFS secondary storage, if the implementation of
> > >> createEntityExtractUrl will involve some code be executed in ssvm to
> > >>copy
> > >> template from the install location to a public accessible web server
> > >> location.
> > >> 	I don't quite understand some of your comments below. This API is not
> > >> used to write any information to S3 bucket/directory. This is used for
> > >> object already existed on S3, and we just provide a URL for user to
> > >> download a template from S3, just like how Amazon provided user a way to
> > >> user to extract a S3 object through generatePresignedUrl. We can discuss
> > >> more on this on collaboration conference.
> > >> 
> > >> 	Thanks	
> > >> 	-min
> > >> 
> > >> 
> > >> 
> > >> On 6/21/13 7:25 AM, "John Burwell" <jb...@basho.com> wrote:
> > >> 
> > >> >Min,
> > >> >
> > >> >(I apologize for my belated reply -- I lost track of this draft in the
> > >> >chaos of the last couple of days.)
> > >> >
> > >> >Upon further review, I think I feel into the confusion between
> > >>management
> > >> >server and ssvm.  This code is executing on the management server side,
> > >> >correct?  Based on my "corrected" understanding is correct, I would
> > >>like
> > >> >to amend my thoughts.  Namely, I would like to see the driver
> > >>operations
> > >> >pushed out to the SSVM where we can use the stream.  As I think about
> > >>it,
> > >> >the management server should not need to interact with the driver.
> > >> >Simply yard up the DataStore attributes + details map and other extract
> > >> >parameters, and send them to the SSVM.  Using this information, the S3
> > >> >driver could open a stream to write the template out to the
> > >> >bucket/directory.  I recognize it changes the protocol between the
> > >> >management server and SSVM, but it simply both sides of the operation
> > >>by
> > >> >allowing the DataStore information to be treated opaquely until it is
> > >> >consumed by the driver to execute the write operation.  I also
> > >>recognize
> > >> >that we may a little late in the cycle to address it for 4.2, and it
> > >>may
> > >> >need to be part of the 4.3 enhancements.
> > >> >
> > >> >Thanks,
> > >> >-John
> > >> >
> > >> >On Jun 18, 2013, at 3:55 PM, Min Chen <mi...@citrix.com> wrote:
> > >> >
> > >> >> John,
> > >> >> 	In that case, how do we keep backward compatibility of
> > >>extractTemplate
> > >> >> api, which requires a URL in the response?
> > >> >> 
> > >> >> 	Thanks
> > >> >> 	-min
> > >> >> 
> > >> >> On 6/18/13 11:53 AM, "John Burwell" <jb...@basho.com> wrote:
> > >> >> 
> > >> >>> Min,
> > >> >>> 
> > >> >>> Looking through the code, I think we can simplify driver operation
> > >>and
> > >> >>> increase robustness by changing
> > >> >>>ImageStoreDriver#createEntityExtractUrl()
> > >> >>> : String to ImageStoreDriver#readEntity(Š) : InputStream.  My first
> > >> >>> concern with the current implementation is that it circumvents any
> > >> >>> connection pooling/resource management underlying client libraries
> > >> >>> provide.  I/O streams provide a higher-level abstraction that allows
> > >> >>> drivers to provide the orchestration components with actual
> > >>resources
> > >> >>> rather String references.  Second, the current interface seems to
> > >> >>>appears
> > >> >>> to assume that an http/https URL will be returned.  With I/O
> > >>streams,
> > >> >>>we
> > >> >>> can support any client library capable of using the standard I/O
> > >> >>> framework -- enabling us to support other protocols for downloading
> > >> >>> templates in the future (e.g. RBD, local filesystem, NBD, etc).
> > >> >>> 
> > >> >>> Thanks,
> > >> >>> -John
> > >> >>> 
> > >> >>> On Jun 18, 2013, at 1:11 PM, Min Chen <mi...@citrix.com> wrote:
> > >> >>> 
> > >> >>>> A new version of using generatePresignedUrl in
> > >>S3ImageStoreDriverImpl
> > >> >>>>is
> > >> >>>> checked into object_store.
> > >> >>>> 
> > >> >>>> THanks
> > >> >>>> -min
> > >> >>>> 
> > >> >>>> On 6/18/13 8:29 AM, "Min Chen" <mi...@citrix.com> wrote:
> > >> >>>> 
> > >> >>>>> Yes, current code is in
> > >> >>>>>S3ImageStoreDriverImpl.createEntityExtractUrl,
> > >> >>>>> which has a security issue mentioned in CLOUDSTACK-3030. I am
> > >>going
> > >> >>>>>to
> > >> >>>>> change it to use generatePresignedUrl api from AWS S3 api.
> > >> >>>>> 
> > >> >>>>> Thanks
> > >> >>>>> -min
> > >> >>>>> 
> > >> >>>>> From: John Burwell <jb...@basho.com>>
> > >> >>>>> Date: Tuesday, June 18, 2013 8:07 AM
> > >> >>>>> To: Min Chen <mi...@citrix.com>>
> > >> >>>>> Cc: Thomas O'Dowd
> > >> >>>>><tp...@cloudian.com>>,
> > >> >>>>> "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> > >> >>>>> <de...@cloudstack.apache.org>>
> > >> >>>>> Subject: Re: Query String Request Authentication(QSRA) support by
> > >>S3
> > >> >>>>> providers
> > >> >>>>> 
> > >> >>>>> Min,
> > >> >>>>> 
> > >> >>>>> Is the code checked into the object_store branch?  If so, which
> > >>lines
> > >> >>>>> in
> > >> >>>>> S3TemplateDownloader?
> > >> >>>>> 
> > >> >>>>> Thanks,
> > >> >>>>> -John
> > >> >>>>> 
> > >> >>>>> On Jun 18, 2013, at 12:39 AM, Min Chen
> > >> >>>>> <mi...@citrix.com>> wrote:
> > >> >>>>> 
> > >> >>>>> Hi John,
> > >> >>>>> 
> > >> >>>>> This is regarding extractTemplate api, where for extractable
> > >> >>>>>template,
> > >> >>>>> users can click "Download Template" button from UI to get a http
> > >>url
> > >> >>>>>to
> > >> >>>>> download the template already stored at S3 without providing S3
> > >> >>>>> credentials. In 4.1, we don't have this issue, since the URL
> > >>returned
> > >> >>>>> is
> > >> >>>>> the public web server location hosted in ssvm, and in 4.2, we are
> > >> >>>>> returning URL pointing to s3 object. Without setting ACL to the S3
> > >> >>>>> object, user cannot directly click the URL returned  from
> > >> >>>>> extractTemplate
> > >> >>>>> api to download the template without providing credentials. By
> > >> >>>>>reading
> > >> >>>>> the AWS SDK doc today, I ran across the following API that I may
> > >>be
> > >> >>>>> able
> > >> >>>>> to use for this purpose:
> > >> >>>>> 
> > >> >>>>> 
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.html?is-ext
> > >>>>>>>er
> > >> >>>>>na
> > >> >>>>> l=
> > >> >>>>> true>     
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>generatePresignedUrl<http://docs.aws.amazon.com/AWSJavaSDK/latest/ja
> > >>>>>>>va
> > >> >>>>>do
> > >> >>>>> c/
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUrl%2
> > >>>>>>>8j
> > >> >>>>>av
> > >> >>>>> a.
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.amazonaws.H
> > >>>>>>>tt
> > >> >>>>>pM
> > >> >>>>> et
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Str
> > >>>>>>>in
> > >> >>>>>g.
> > >> >>>>> ht
> > >> >>>>> ml?is-external=true> bucketName,
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html
> > >>>>>>>?i
> > >> >>>>>s-
> > >> >>>>> ex
> > >> >>>>> ternal=true> key,
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html?is-
> > >>>>>>>ex
> > >> >>>>>te
> > >> >>>>> rn
> > >> >>>>> al=true> expiration,
> > >> >>>>> 
> > >> >>>>> 
> > >> 
> > >>>>>>>HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/
> > >>>>>>>am
> > >> >>>>>az
> > >> >>>>> on
> > >> >>>>> aws/HttpMethod.html> method)
> > >> >>>>>         Returns a pre-signed URL for accessing an Amazon S3
> > >>resource.
> > >> >>>>> 
> > >> >>>>> This is along the same line as QSRA mentioned by Tom, by wrapped
> > >>in
> > >> >>>>> AmazonS3Client for easy consumption. By using this method, I think
> > >> >>>>> that I
> > >> >>>>> don't need to change ACL of S3 object to open a security hole.
> > >> >>>>> 
> > >> >>>>> Thanks
> > >> >>>>> -min
> > >> >>>>> 
> > >> >>>>> From: John Burwell <jb...@basho.com>>
> > >> >>>>> Date: Monday, June 17, 2013 7:38 PM
> > >> >>>>> To: Min Chen <mi...@citrix.com>>
> > >> >>>>> Cc: Thomas O'Dowd
> > >> >>>>><tp...@cloudian.com>>,
> > >> >>>>> "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> > >> >>>>> <de...@cloudstack.apache.org>>
> > >> >>>>> Subject: Re: Query String Request Authentication(QSRA) support by
> > >>S3
> > >> >>>>> providers
> > >> >>>>> 
> > >> >>>>> Min,
> > >> >>>>> 
> > >> >>>>> Why are we mucking with ACLs at all?  The best security practice
> > >> >>>>>would
> > >> >>>>> be
> > >> >>>>> to create a bucket for CloudStack's use and assign it a dedicated
> > >> >>>>> access
> > >> >>>>> key and secret key pair with read/write access only to that
> > >>bucket.
> > >> >>>>> Requiring an administrative account to an object store opens an
> > >> >>>>> unnecessarily large attack surface.  Therefore, as implemented in
> > >> >>>>>4.1,
> > >> >>>>> we
> > >> >>>>> should defer bucket creation, ACL assignment, and credential
> > >>creation
> > >> >>>>> to
> > >> >>>>> the administrator/operator.
> > >> >>>>> 
> > >> >>>>> Thanks,
> > >> >>>>> -John
> > >> >>>>> 
> > >> >>>>> On Jun 17, 2013, at 1:15 PM, Min Chen
> > >> >>>>> <mi...@citrix.com>> wrote:
> > >> >>>>> 
> > >> >>>>> Tom filed a very good bug for ACL setting change on S3 object when
> > >> >>>>> users
> > >> >>>>> issue extractTemplate API
> > >> >>>>> (https://issues.apache.org/jira/browse/CLOUDSTACK-3030), and his
> > >> >>>>> recommendation of using Query String Request Authentication (QSRA)
> > >> >>>>> alternative sounds like a right approach to fix this bug. Before
> > >> >>>>> implementing it, I would like to confirm if QSRA should be
> > >>supported
> > >> >>>>>by
> > >> >>>>> all S3 providers if they claim that they are AWS s3 compatible. If
> > >> >>>>>so,
> > >> >>>>> we
> > >> >>>>> will make this assumption in our code. Based on Tom, Cloudian is
> > >> >>>>> supporting it. How about RiakCS, John?
> > >> >>>>> 
> > >> >>>>> Thanks
> > >> >>>>> -min
> > >> >>>>> 
> > >> >>>>> 
> > >> >>>> 
> > >> >>> 
> > >> >> 
> > >> >
> > >> 
> > >
> > >-- 
> > >Cloudian KK - http://www.cloudian.com/get-started.html
> > >Fancy 100TB of full featured S3 Storage?
> > >Checkout the Cloudian(R) Community Edition!
> > >
> > 
> 

-- 
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 100TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!

RE: Query String Request Authentication(QSRA) support by S3 providers

Posted by Edison Su <Ed...@citrix.com>.

Hi Thomas, as Min will OOO in the next week, would you mind to pick up some bugs related to S3?:)

> -----Original Message-----
> From: Min Chen [mailto:min.chen@citrix.com]
> Sent: Wednesday, July 03, 2013 5:23 PM
> To: Thomas O'Dowd
> Cc: dev@cloudstack.apache.org; Jessica Wang
> Subject: Re: Query String Request Authentication(QSRA) support by S3
> providers
> 
> Hi Tom,
> 
> 	I can reproduce this issue using Cloudian, after investigation, I
> realized that this is a bug in Amazon SDK we have used, based on this
> thread:
> http://stackoverflow.com/questions/15473582/amazon-s3-presigned-urls-
> escape
> -the-slashes-in-the-key. When generatePresignedUrl is called it takes the
> entire key and escapes it, and then creates a signature using the escaped key.
> You cannot use the signature from the escaped key and combine it with the
> unescaped key in the URL. See the bug code here:
> 
> 	String resourcePath = "/" +
> 	((bucketName != null) ? bucketName + "/" : "") +
> 	((key != null) ? ServiceUtils.urlEncode(key) : "") +
> 	((subResource != null) ? "?" + subResource : "");
> 
> We have two options to fix this:
> 	1. Either upgrade Amazon SDK to use 1.4.3 version, someone in that
> thread claimed that it is fixed in that version, but I haven't checked that.
> Currently CloudStack is using 1.3.21. Not sure if this will break CloudStack
> cloud_bridge.
> 	2. Workaround by creating customized AmazonS3Client to change
> the internal implementation on this.
> 
> 	Thanks
> 	-min
> 
> 
> 
> On 7/2/13 11:31 PM, "Thomas O'Dowd" <tp...@cloudian.com> wrote:
> 
> >Excellent. The link is there now. Thank you Min. I verified that bug
> >and closed it.
> >
> >However - now that I can finally click the download link... I ran into
> >the issue that the link doesn't work on AWS or Cloudian. Please see
> >this bug for details (latest 4.2 updates included in my test).
> >
> >    https://issues.apache.org/jira/browse/CLOUDSTACK-3341
> >
> >Tom.
> >
> >On Tue, 2013-07-02 at 22:54 +0000, Min Chen wrote:
> >> Hi Tom,
> >> 	I investigated this issue through the db dump you provided in the
> >> bug, this is an issue with our db view template_view creation script,
> >> and it has been fixed in resolving another bug
> >> (https://issues.apache.org/jira/browse/CLOUDSTACK-3314). I have
> >> verified the fix using your db dump on my local setup. Please check
> >> out latest 4.2 or master code to try again.
> >>
> >> 	Thanks
> >> 	-min
> >>
> >> On 7/2/13 2:18 PM, "Min Chen" <mi...@citrix.com> wrote:
> >>
> >> >Tom, this seems like an issue with entry stored in our DB. I will
> >> >take
> >>a
> >> >look at this bug and update you. Just to clarify, this symptom only
> >> >happens when you register these templates to Amazon S3, not for
> >>Cloudian
> >> >or RiakCS S3, right?
> >> >
> >> >Thanks
> >> >-min
> >> >
> >> >On 7/1/13 7:27 PM, "Thomas O'Dowd" <tp...@cloudian.com> wrote:
> >> >
> >> >>Yes thanks Jessica. I re-opened the bug again. I know its not a gui
> >> >>problem per-say in that the template is not ready to show the
> >> >>download link. However, it never becomes ready is the actual
> >> >>problem. What sets the "isready" property to true? As far as I can
> >> >>see, the objects in
> >>the
> >> >>S3 stores (AWS or Cloudian) are complete and from my perspective
> >>"ready"
> >> >>to download/use. It sounds like a bug when registering the template.
> >> >>
> >> >>Tom.
> >> >>
> >> >>On Mon, 2013-07-01 at 18:54 +0000, Jessica Wang wrote:
> >> >>> Thomas,
> >> >>>
> >> >>> I checked the data you provided.
> >> >>>
> >> >>> The reason that the 2 templates("MyTiny", "AnotherTiny") have no
> >> >>>download button is because they are not ready  (i.e. their
> >> >>>"isready" property is false).
> >> >>>
> >> >>> Download button is only available when "isready" property is true.
> >> >>>
> >> >>> Jessica
> >> >>>
> >> >>> -----Original Message-----
> >> >>> From: Thomas O'Dowd [mailto:tpodowd@cloudian.com]
> >> >>> Sent: Thursday, June 27, 2013 8:04 PM
> >> >>> To: Min Chen
> >> >>> Cc: dev@cloudstack.apache.org; Jessica Wang
> >> >>> Subject: Re: Query String Request Authentication(QSRA) support by
> >> >>>S3 providers
> >> >>>
> >> >>> Hi Min/Jessica,
> >> >>>
> >> >>> I attached an image to that issue to show what what my browser is
> >> >>> showing.
> >> >>>
> >> >>>     https://issues.apache.org/jira/browse/CLOUDSTACK-3220
> >> >>>
> >> >>> Tom.
> >> >>>
> >> >>> On Fri, 2013-06-28 at 09:45 +0900, Thomas O'Dowd wrote:
> >> >>> > Hi Min,
> >> >>> >
> >> >>> > Yes. I'll try it again today to check again but when I added
> >>Amazon
> >> >>>S3
> >> >>> > as the S3 secondary storage and uploaded a template, I was not
> >>shown
> >> >>>the
> >> >>> > "download template" link. However - for Cloudian S3, I am shown
> >>it so
> >> >>> > I'm wondering why.
> >> >>> >
> >> >>> > Tom.
> >> >>> >
> >> >>> > On Fri, 2013-06-28 at 00:26 +0000, Min Chen wrote:
> >> >>> > > Hi Tom,
> >> >>> > >
> >> >>> > > 	Are you saying that you cannot see a Download Template
> >> >>> > > button
> >>from
> >> >>>UI
> >> >>> > > when Amazon S3 is added as secondary storage? I only tested
> >> >>> > > with
> >> >>>RiakCS
> >> >>> > > and Cloudian, so didn't see this issue. But I am CC Jessica
> >> >>> > > her
> >>to
> >> >>>confirm
> >> >>> > > what special handling is done in UI to enable/disable a
> >> >>> > > button
> >>from
> >> >>>UI.
> >> >>> > >
> >> >>> > > 	Thanks
> >> >>> > > 	-min
> >> >>> > >
> >> >>> > > On 6/27/13 5:23 PM, "Thomas O'Dowd" <tp...@cloudian.com>
> >>wrote:
> >> >>> > >
> >> >>> > > >Hi Min,
> >> >>> > > >
> >> >>> > > >Can you check this bug? I'm trying to test this feature for
> >>Amazon
> >> >>>but
> >> >>> > > >having no luck getting the Download template link/button to
> >> >>>appear.
> >> >>> > > >
> >> >>> > > >https://issues.apache.org/jira/browse/CLOUDSTACK-3220
> >> >>> > > >
> >> >>> > > >Thanks,
> >> >>> > > >
> >> >>> > > >Tom.
> >> >>> > > >
> >> >>> > > >On Fri, 2013-06-21 at 17:21 +0000, Min Chen wrote:
> >> >>> > > >> John,
> >> >>> > > >>
> >> >>> > > >> 	For S3, the api call createEntityExtractUrl is done on
> >> >>>management
> >> >>> > > >>server
> >> >>> > > >> side; while for NFS secondary storage, if the
> >> >>> > > >>implementation
> >>of
> >> >>> > > >> createEntityExtractUrl will involve some code be executed
> >> >>> > > >> in
> >> >>>ssvm to
> >> >>> > > >>copy
> >> >>> > > >> template from the install location to a public accessible
> >> >>> > > >>web
> >> >>>server
> >> >>> > > >> location.
> >> >>> > > >> 	I don't quite understand some of your comments below.
> >> >>> > > >> This
> >>API
> >> >>>is not
> >> >>> > > >> used to write any information to S3 bucket/directory. This
> >> >>> > > >> is
> >> >>>used for
> >> >>> > > >> object already existed on S3, and we just provide a URL
> >> >>> > > >> for
> >>user
> >> >>>to
> >> >>> > > >> download a template from S3, just like how Amazon provided
> >>user
> >> >>>a way to
> >> >>> > > >> user to extract a S3 object through generatePresignedUrl.
> >> >>> > > >> We
> >>can
> >> >>>discuss
> >> >>> > > >> more on this on collaboration conference.
> >> >>> > > >>
> >> >>> > > >> 	Thanks
> >> >>> > > >> 	-min
> >> >>> > > >>
> >> >>> > > >>
> >> >>> > > >>
> >> >>> > > >> On 6/21/13 7:25 AM, "John Burwell" <jb...@basho.com>
> >>wrote:
> >> >>> > > >>
> >> >>> > > >> >Min,
> >> >>> > > >> >
> >> >>> > > >> >(I apologize for my belated reply -- I lost track of this
> >>draft
> >> >>>in the
> >> >>> > > >> >chaos of the last couple of days.)
> >> >>> > > >> >
> >> >>> > > >> >Upon further review, I think I feel into the confusion
> >>between
> >> >>> > > >>management
> >> >>> > > >> >server and ssvm.  This code is executing on the
> >> >>> > > >> >management
> >> >>>server side,
> >> >>> > > >> >correct?  Based on my "corrected" understanding is
> >> >>> > > >> >correct,
> >>I
> >> >>>would
> >> >>> > > >>like
> >> >>> > > >> >to amend my thoughts.  Namely, I would like to see the
> >>driver
> >> >>> > > >>operations
> >> >>> > > >> >pushed out to the SSVM where we can use the stream.  As I
> >>think
> >> >>>about
> >> >>> > > >>it,
> >> >>> > > >> >the management server should not need to interact with
> >> >>> > > >> >the
> >> >>>driver.
> >> >>> > > >> >Simply yard up the DataStore attributes + details map and
> >>other
> >> >>>extract
> >> >>> > > >> >parameters, and send them to the SSVM.  Using this
> >>information,
> >> >>>the S3
> >> >>> > > >> >driver could open a stream to write the template out to
> >> >>> > > >> >the bucket/directory.  I recognize it changes the
> >> >>> > > >> >protocol
> >>between
> >> >>>the
> >> >>> > > >> >management server and SSVM, but it simply both sides of
> >> >>> > > >> >the
> >> >>>operation
> >> >>> > > >>by
> >> >>> > > >> >allowing the DataStore information to be treated opaquely
> >>until
> >> >>>it is
> >> >>> > > >> >consumed by the driver to execute the write operation.  I
> >>also
> >> >>> > > >>recognize
> >> >>> > > >> >that we may a little late in the cycle to address it for
> >>4.2,
> >> >>>and it
> >> >>> > > >>may
> >> >>> > > >> >need to be part of the 4.3 enhancements.
> >> >>> > > >> >
> >> >>> > > >> >Thanks,
> >> >>> > > >> >-John
> >> >>> > > >> >
> >> >>> > > >> >On Jun 18, 2013, at 3:55 PM, Min Chen
> >> >>> > > >> ><mi...@citrix.com>
> >> >>>wrote:
> >> >>> > > >> >
> >> >>> > > >> >> John,
> >> >>> > > >> >> 	In that case, how do we keep backward compatibility
> of
> >> >>> > > >>extractTemplate
> >> >>> > > >> >> api, which requires a URL in the response?
> >> >>> > > >> >>
> >> >>> > > >> >> 	Thanks
> >> >>> > > >> >> 	-min
> >> >>> > > >> >>
> >> >>> > > >> >> On 6/18/13 11:53 AM, "John Burwell"
> >> >>> > > >> >> <jb...@basho.com>
> >> >>>wrote:
> >> >>> > > >> >>
> >> >>> > > >> >>> Min,
> >> >>> > > >> >>>
> >> >>> > > >> >>> Looking through the code, I think we can simplify
> >> >>> > > >> >>> driver
> >> >>>operation
> >> >>> > > >>and
> >> >>> > > >> >>> increase robustness by changing
> >> >>> > > >> >>>ImageStoreDriver#createEntityExtractUrl()
> >> >>> > > >> >>> : String to ImageStoreDriver#readEntity(Š) : InputStream.
> >> >>>My first
> >> >>> > > >> >>> concern with the current implementation is that it
> >> >>>circumvents any
> >> >>> > > >> >>> connection pooling/resource management underlying
> >> >>> > > >> >>> client
> >> >>>libraries
> >> >>> > > >> >>> provide.  I/O streams provide a higher-level
> >> >>> > > >> >>> abstraction
> >> >>>that allows
> >> >>> > > >> >>> drivers to provide the orchestration components with
> >>actual
> >> >>> > > >>resources
> >> >>> > > >> >>> rather String references.  Second, the current
> >> >>> > > >> >>> interface
> >> >>>seems to
> >> >>> > > >> >>>appears
> >> >>> > > >> >>> to assume that an http/https URL will be returned.
> >> >>> > > >> >>>With
> >>I/O
> >> >>> > > >>streams,
> >> >>> > > >> >>>we
> >> >>> > > >> >>> can support any client library capable of using the
> >>standard
> >> >>>I/O
> >> >>> > > >> >>> framework -- enabling us to support other protocols
> >> >>> > > >> >>> for
> >> >>>downloading
> >> >>> > > >> >>> templates in the future (e.g. RBD, local filesystem,
> >> >>> > > >> >>> NBD,
> >> >>>etc).
> >> >>> > > >> >>>
> >> >>> > > >> >>> Thanks,
> >> >>> > > >> >>> -John
> >> >>> > > >> >>>
> >> >>> > > >> >>> On Jun 18, 2013, at 1:11 PM, Min Chen
> >><mi...@citrix.com>
> >> >>>wrote:
> >> >>> > > >> >>>
> >> >>> > > >> >>>> A new version of using generatePresignedUrl in
> >> >>> > > >>S3ImageStoreDriverImpl
> >> >>> > > >> >>>>is
> >> >>> > > >> >>>> checked into object_store.
> >> >>> > > >> >>>>
> >> >>> > > >> >>>> THanks
> >> >>> > > >> >>>> -min
> >> >>> > > >> >>>>
> >> >>> > > >> >>>> On 6/18/13 8:29 AM, "Min Chen" <mi...@citrix.com>
> >>wrote:
> >> >>> > > >> >>>>
> >> >>> > > >> >>>>> Yes, current code is in
> >> >>> > > >> >>>>>S3ImageStoreDriverImpl.createEntityExtractUrl,
> >> >>> > > >> >>>>> which has a security issue mentioned in
> >>CLOUDSTACK-3030. I
> >> >>>am
> >> >>> > > >>going
> >> >>> > > >> >>>>>to
> >> >>> > > >> >>>>> change it to use generatePresignedUrl api from AWS
> >> >>> > > >> >>>>>S3
> >>api.
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Thanks
> >> >>> > > >> >>>>> -min
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> From: John Burwell
> >> >>><jb...@basho.com>>
> >> >>> > > >> >>>>> Date: Tuesday, June 18, 2013 8:07 AM
> >> >>> > > >> >>>>> To: Min Chen
> >> >>><mi...@citrix.com>>
> >> >>> > > >> >>>>> Cc: Thomas O'Dowd
> >> >>> > > >> >>>>><tpodowd@cloudian.com<mailto:tpodowd@cloudian.c
> om>>,
> >> >>> > > >> >>>>>
> >> >>>"dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> >> >>> > > >> >>>>>
> >> >>><de...@cloudstack.apache.org>>
> >> >>> > > >> >>>>> Subject: Re: Query String Request
> >> >>> > > >> >>>>> Authentication(QSRA)
> >> >>>support by
> >> >>> > > >>S3
> >> >>> > > >> >>>>> providers
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Min,
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Is the code checked into the object_store branch?
> >> >>> > > >> >>>>> If
> >>so,
> >> >>>which
> >> >>> > > >>lines
> >> >>> > > >> >>>>> in
> >> >>> > > >> >>>>> S3TemplateDownloader?
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Thanks,
> >> >>> > > >> >>>>> -John
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> On Jun 18, 2013, at 12:39 AM, Min Chen
> >> >>> > > >> >>>>> <mi...@citrix.com>>
> >>wrote:
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Hi John,
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> This is regarding extractTemplate api, where for
> >> >>>extractable
> >> >>> > > >> >>>>>template,
> >> >>> > > >> >>>>> users can click "Download Template" button from UI
> >> >>> > > >> >>>>>to
> >>get
> >> >>>a http
> >> >>> > > >>url
> >> >>> > > >> >>>>>to
> >> >>> > > >> >>>>> download the template already stored at S3 without
> >> >>>providing S3
> >> >>> > > >> >>>>> credentials. In 4.1, we don't have this issue, since
> >>the
> >> >>>URL
> >> >>> > > >>returned
> >> >>> > > >> >>>>> is
> >> >>> > > >> >>>>> the public web server location hosted in ssvm, and
> >> >>> > > >> >>>>> in
> >>4.2,
> >> >>>we are
> >> >>> > > >> >>>>> returning URL pointing to s3 object. Without setting
> >>ACL
> >> >>>to the S3
> >> >>> > > >> >>>>> object, user cannot directly click the URL returned
> >>from
> >> >>> > > >> >>>>> extractTemplate
> >> >>> > > >> >>>>> api to download the template without providing
> >> >>>credentials. By
> >> >>> > > >> >>>>>reading
> >> >>> > > >> >>>>> the AWS SDK doc today, I ran across the following
> >> >>> > > >> >>>>>API
> >>that
> >> >>>I may
> >> >>> > > >>be
> >> >>> > > >> >>>>> able
> >> >>> > > >> >>>>> to use for this purpose:
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.
> htm
> >>>>>>>>>>>>l?i
> >>>>>>>>>>>>s-
> >> >>>>>>>>>>e
> >> >>>>>>>>>>xt
> >> >>> > > >>>>>>>er
> >> >>> > > >> >>>>>na
> >> >>> > > >> >>>>> l=
> >> >>> > > >> >>>>> true>
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>generatePresignedUrl<http://docs.aws.amazon.com/AWSJa
> vaSDK/l
> >>>>>>>>>>>>ate
> >>>>>>>>>>>>st
> >> >>>>>>>>>>/
> >> >>>>>>>>>>ja
> >> >>> > > >>>>>>>va
> >> >>> > > >> >>>>>do
> >> >>> > > >> >>>>> c/
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>com/amazonaws/services/s3/AmazonS3Client.html#generat
> ePresig
> >>>>>>>>>>>>ned
> >>>>>>>>>>>>Ur
> >> >>>>>>>>>>l
> >> >>>>>>>>>>%2
> >> >>> > > >>>>>>>8j
> >> >>> > > >> >>>>>av
> >> >>> > > >> >>>>> a.
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.am
> a
> >>>>>>>>>>>>zon
> >>>>>>>>>>>>aw
> >> >>>>>>>>>>s
> >> >>>>>>>>>>.H
> >> >>> > > >>>>>>>tt
> >> >>> > > >> >>>>>pM
> >> >>> > > >> >>>>> et
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/ja
> va/
> >>>>>>>>>>>>lan
> >>>>>>>>>>>>g/
> >> >>>>>>>>>>S
> >> >>>>>>>>>>tr
> >> >>> > > >>>>>>>in
> >> >>> > > >> >>>>>g.
> >> >>> > > >> >>>>> ht
> >> >>> > > >> >>>>> ml?is-external=true> bucketName,
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/St
> r
> >>>>>>>>>>>>ing
> >>>>>>>>>>>>.h
> >> >>>>>>>>>>t
> >> >>>>>>>>>>ml
> >> >>> > > >>>>>>>?i
> >> >>> > > >> >>>>>s-
> >> >>> > > >> >>>>> ex
> >> >>> > > >> >>>>> ternal=true> key,
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Dat
> e.
> >>>>>>>>>>>>htm
> >>>>>>>>>>>>l?
> >> >>>>>>>>>>i
> >> >>>>>>>>>>s-
> >> >>> > > >>>>>>>ex
> >> >>> > > >> >>>>>te
> >> >>> > > >> >>>>> rn
> >> >>> > > >> >>>>> al=true> expiration,
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >>
> >> >>> > >
> >>
> >>>>>>>>>>>>HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/lat
> est/java
> >>>>>>>>>>>>doc
> >>>>>>>>>>>>/c
> >> >>>>>>>>>>o
> >> >>>>>>>>>>m/
> >> >>> > > >>>>>>>am
> >> >>> > > >> >>>>>az
> >> >>> > > >> >>>>> on
> >> >>> > > >> >>>>> aws/HttpMethod.html> method)
> >> >>> > > >> >>>>>         Returns a pre-signed URL for accessing an
> >>Amazon
> >> >>>S3
> >> >>> > > >>resource.
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> This is along the same line as QSRA mentioned by
> >> >>> > > >> >>>>> Tom,
> >>by
> >> >>>wrapped
> >> >>> > > >>in
> >> >>> > > >> >>>>> AmazonS3Client for easy consumption. By using this
> >>method,
> >> >>>I think
> >> >>> > > >> >>>>> that I
> >> >>> > > >> >>>>> don't need to change ACL of S3 object to open a
> >>security
> >> >>>hole.
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Thanks
> >> >>> > > >> >>>>> -min
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> From: John Burwell
> >> >>><jb...@basho.com>>
> >> >>> > > >> >>>>> Date: Monday, June 17, 2013 7:38 PM
> >> >>> > > >> >>>>> To: Min Chen
> >> >>><mi...@citrix.com>>
> >> >>> > > >> >>>>> Cc: Thomas O'Dowd
> >> >>> > > >> >>>>><tpodowd@cloudian.com<mailto:tpodowd@cloudian.c
> om>>,
> >> >>> > > >> >>>>>
> >> >>>"dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> >> >>> > > >> >>>>>
> >> >>><de...@cloudstack.apache.org>>
> >> >>> > > >> >>>>> Subject: Re: Query String Request
> >> >>> > > >> >>>>> Authentication(QSRA)
> >> >>>support by
> >> >>> > > >>S3
> >> >>> > > >> >>>>> providers
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Min,
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Why are we mucking with ACLs at all?  The best
> >> >>> > > >> >>>>> security
> >> >>>practice
> >> >>> > > >> >>>>>would
> >> >>> > > >> >>>>> be
> >> >>> > > >> >>>>> to create a bucket for CloudStack's use and assign
> >> >>> > > >> >>>>>it a
> >> >>>dedicated
> >> >>> > > >> >>>>> access
> >> >>> > > >> >>>>> key and secret key pair with read/write access only
> >> >>> > > >> >>>>> to
> >> >>>that
> >> >>> > > >>bucket.
> >> >>> > > >> >>>>> Requiring an administrative account to an object
> >> >>> > > >> >>>>> store
> >> >>>opens an
> >> >>> > > >> >>>>> unnecessarily large attack surface.  Therefore, as
> >> >>>implemented in
> >> >>> > > >> >>>>>4.1,
> >> >>> > > >> >>>>> we
> >> >>> > > >> >>>>> should defer bucket creation, ACL assignment, and
> >> >>>credential
> >> >>> > > >>creation
> >> >>> > > >> >>>>> to
> >> >>> > > >> >>>>> the administrator/operator.
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Thanks,
> >> >>> > > >> >>>>> -John
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> On Jun 17, 2013, at 1:15 PM, Min Chen
> >> >>> > > >> >>>>> <mi...@citrix.com>>
> >>wrote:
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Tom filed a very good bug for ACL setting change on
> >> >>> > > >> >>>>> S3
> >> >>>object when
> >> >>> > > >> >>>>> users
> >> >>> > > >> >>>>> issue extractTemplate API
> >> >>> > > >> >>>>>
> >>(https://issues.apache.org/jira/browse/CLOUDSTACK-3030),
> >> >>>and his
> >> >>> > > >> >>>>> recommendation of using Query String Request
> >> >>>Authentication (QSRA)
> >> >>> > > >> >>>>> alternative sounds like a right approach to fix this
> >>bug.
> >> >>>Before
> >> >>> > > >> >>>>> implementing it, I would like to confirm if QSRA
> >>should be
> >> >>> > > >>supported
> >> >>> > > >> >>>>>by
> >> >>> > > >> >>>>> all S3 providers if they claim that they are AWS s3
> >> >>>compatible. If
> >> >>> > > >> >>>>>so,
> >> >>> > > >> >>>>> we
> >> >>> > > >> >>>>> will make this assumption in our code. Based on Tom,
> >> >>>Cloudian is
> >> >>> > > >> >>>>> supporting it. How about RiakCS, John?
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>> Thanks
> >> >>> > > >> >>>>> -min
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>>
> >> >>> > > >> >>>>
> >> >>> > > >> >>>
> >> >>> > > >> >>
> >> >>> > > >> >
> >> >>> > > >>
> >> >>> > > >
> >> >>> > > >--
> >> >>> > > >Cloudian KK - http://www.cloudian.com/get-started.html
> >> >>> > > >Fancy 100TB of full featured S3 Storage?
> >> >>> > > >Checkout the Cloudian(R) Community Edition!
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >> >>
> >> >>--
> >> >>Cloudian KK - http://www.cloudian.com/get-started.html
> >> >>Fancy 100TB of full featured S3 Storage?
> >> >>Checkout the Cloudian(R) Community Edition!
> >> >>
> >> >
> >>
> >
> >--
> >Cloudian KK - http://www.cloudian.com/get-started.html
> >Fancy 100TB of full featured S3 Storage?
> >Checkout the Cloudian(R) Community Edition!
> >

Re: AWS SDK version for object store

Posted by Chip Childers <ch...@sungard.com>.

On Mon, Jul 8, 2013 at 5:26 AM, Donal Lafferty
<do...@citrix.com> wrote:
> +1

+1 from me as well.

>
> IMHO, trade off is between getting fixes, especially security fixes, and possibility that AWS introduce a tweak that breaks our use case.
>
> The subtlety is that you'd use the latest version before integration testing rather than before release.  I.e. get as many fixes as possible, but also make sure we don't ship something broken.
>
> Am I splitting hairs?  (http://idioms.thefreedictionary.com/split+hairs)
>
> DL
>
>
>> -----Original Message-----
>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>> Sent: 06 July 2013 2:44 PM
>> To: dev@cloudstack.apache.org
>> Subject: Re: AWS SDK version for object store
>>
>> On Fri, Jul 05, 2013 at 06:27:46PM +0530, Prasanna Santhanam wrote:
>> > There was another issue I found today with the SDK v1.3.21. It appears
>> > I can't use periods in the name of a bucket. I see an SSL Exception
>> > when this is done [1]. The bug is fixed in the v1.3.22. There are also
>> > couple of other issues fixed related to s3 on the latest version of
>> > the SDK.  Is there any reason not to use the latest stable SDK version
>> > in CloudStack? IOW, why did we choose the v1.3.21?
>> >
>> > [1] http://issues.apache.org/jira/browse/CLOUDSTACK-3374
>> >
>> Fixed by bumping up the version to 1.3.22. But would still like to bump it up
>> to the latest stable version of the SDK available at the time of the release ACS
>> 4.2
>>
>> Thoughts?
>>
>>
>> --
>> Prasanna.,
>>
>> ------------------------
>> Powered by BigRock.com
>
>

RE: AWS SDK version for object store

Posted by Donal Lafferty <do...@citrix.com>.

+1

IMHO, trade off is between getting fixes, especially security fixes, and possibility that AWS introduce a tweak that breaks our use case.

The subtlety is that you'd use the latest version before integration testing rather than before release.  I.e. get as many fixes as possible, but also make sure we don't ship something broken.

Am I splitting hairs?  (http://idioms.thefreedictionary.com/split+hairs) 

DL


> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: 06 July 2013 2:44 PM
> To: dev@cloudstack.apache.org
> Subject: Re: AWS SDK version for object store
> 
> On Fri, Jul 05, 2013 at 06:27:46PM +0530, Prasanna Santhanam wrote:
> > There was another issue I found today with the SDK v1.3.21. It appears
> > I can't use periods in the name of a bucket. I see an SSL Exception
> > when this is done [1]. The bug is fixed in the v1.3.22. There are also
> > couple of other issues fixed related to s3 on the latest version of
> > the SDK.  Is there any reason not to use the latest stable SDK version
> > in CloudStack? IOW, why did we choose the v1.3.21?
> >
> > [1] http://issues.apache.org/jira/browse/CLOUDSTACK-3374
> >
> Fixed by bumping up the version to 1.3.22. But would still like to bump it up
> to the latest stable version of the SDK available at the time of the release ACS
> 4.2
> 
> Thoughts?
> 
> 
> --
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com

Re: AWS SDK version for object store

Posted by Prasanna Santhanam <ts...@apache.org>.

On Fri, Jul 05, 2013 at 06:27:46PM +0530, Prasanna Santhanam wrote:
> There was another issue I found today with the SDK v1.3.21. It appears
> I can't use periods in the name of a bucket. I see an SSL Exception
> when this is done [1]. The bug is fixed in the v1.3.22. There are also
> couple of other issues fixed related to s3 on the latest version of
> the SDK.  Is there any reason not to use the latest stable SDK version
> in CloudStack? IOW, why did we choose the v1.3.21?
> 
> [1] http://issues.apache.org/jira/browse/CLOUDSTACK-3374
> 
Fixed by bumping up the version to 1.3.22. But would still like to
bump it up to the latest stable version of the SDK available at the
time of the release ACS 4.2

Thoughts?


-- 
Prasanna.,

------------------------
Powered by BigRock.com

Re: AWS SDK version for object store

Posted by Thomas O'Dowd <tp...@cloudian.com>.

I'm +1 also to taking the latest stable release.

Tom.

On Mon, 2013-07-08 at 13:28 -0400, John Burwell wrote:
> Edison,
> 
> I vote to take the latest stable from Amazon.  In addition to keeping up with security and bug fixes, it also ensures that we stay in sync with any "refinements" that Amazon introduces as part of their continuous improvement/deployment model.
> 
> Thanks,
> -John
> 
> On Jul 8, 2013, at 1:26 PM, Edison Su <Ed...@citrix.com> wrote:
> 
> > From CloudStack point of view, there should be minor changes in order to adapt to latest AWS SDK. What's your guys option, John, Thomas? Is it ok to upgrade to latest SDK?
> > 
> >> -----Original Message-----
> >> From: Prasanna Santhanam [mailto:tsp@apache.org]
> >> Sent: Friday, July 05, 2013 5:58 AM
> >> To: dev@cloudstack.apache.org
> >> Subject: AWS SDK version for object store
> >> 
> >> There was another issue I found today with the SDK v1.3.21. It appears I can't
> >> use periods in the name of a bucket. I see an SSL Exception when this is done
> >> [1]. The bug is fixed in the v1.3.22. There are also couple of other issues fixed
> >> related to s3 on the latest version of the SDK.  Is there any reason not to use
> >> the latest stable SDK version in CloudStack? IOW, why did we choose the
> >> v1.3.21?
> >> 
> >> [1] http://issues.apache.org/jira/browse/CLOUDSTACK-3374
> >> 
> >> --
> >> Prasanna.,
> >> 
> >> ------------------------
> >> Powered by BigRock.com
> > 
> 

-- 
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 100TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!

Re: AWS SDK version for object store

Posted by John Burwell <jb...@basho.com>.

Edison,

I vote to take the latest stable from Amazon.  In addition to keeping up with security and bug fixes, it also ensures that we stay in sync with any "refinements" that Amazon introduces as part of their continuous improvement/deployment model.

Thanks,
-John

On Jul 8, 2013, at 1:26 PM, Edison Su <Ed...@citrix.com> wrote:

> From CloudStack point of view, there should be minor changes in order to adapt to latest AWS SDK. What's your guys option, John, Thomas? Is it ok to upgrade to latest SDK?
> 
>> -----Original Message-----
>> From: Prasanna Santhanam [mailto:tsp@apache.org]
>> Sent: Friday, July 05, 2013 5:58 AM
>> To: dev@cloudstack.apache.org
>> Subject: AWS SDK version for object store
>> 
>> There was another issue I found today with the SDK v1.3.21. It appears I can't
>> use periods in the name of a bucket. I see an SSL Exception when this is done
>> [1]. The bug is fixed in the v1.3.22. There are also couple of other issues fixed
>> related to s3 on the latest version of the SDK.  Is there any reason not to use
>> the latest stable SDK version in CloudStack? IOW, why did we choose the
>> v1.3.21?
>> 
>> [1] http://issues.apache.org/jira/browse/CLOUDSTACK-3374
>> 
>> --
>> Prasanna.,
>> 
>> ------------------------
>> Powered by BigRock.com
>

RE: AWS SDK version for object store

Posted by Edison Su <Ed...@citrix.com>.

>From CloudStack point of view, there should be minor changes in order to adapt to latest AWS SDK. What's your guys option, John, Thomas? Is it ok to upgrade to latest SDK?

> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: Friday, July 05, 2013 5:58 AM
> To: dev@cloudstack.apache.org
> Subject: AWS SDK version for object store
> 
> There was another issue I found today with the SDK v1.3.21. It appears I can't
> use periods in the name of a bucket. I see an SSL Exception when this is done
> [1]. The bug is fixed in the v1.3.22. There are also couple of other issues fixed
> related to s3 on the latest version of the SDK.  Is there any reason not to use
> the latest stable SDK version in CloudStack? IOW, why did we choose the
> v1.3.21?
> 
> [1] http://issues.apache.org/jira/browse/CLOUDSTACK-3374
> 
> --
> Prasanna.,
> 
> ------------------------
> Powered by BigRock.com

AWS SDK version for object store

Posted by Prasanna Santhanam <ts...@apache.org>.

There was another issue I found today with the SDK v1.3.21. It appears
I can't use periods in the name of a bucket. I see an SSL Exception
when this is done [1]. The bug is fixed in the v1.3.22. There are also
couple of other issues fixed related to s3 on the latest version of
the SDK.  Is there any reason not to use the latest stable SDK version
in CloudStack? IOW, why did we choose the v1.3.21?

[1] http://issues.apache.org/jira/browse/CLOUDSTACK-3374

-- 
Prasanna.,

------------------------
Powered by BigRock.com

RE: Query String Request Authentication(QSRA) support by S3 providers

Posted by Jessica Wang <Je...@citrix.com>.

Sanjeev,

Thanks for the confirmation.

Jessica

-----Original Message-----
From: Sanjeev Neelarapu 
Sent: Thursday, July 18, 2013 11:05 PM
To: dev@cloudstack.apache.org; Jessica Wang
Cc: Edison Su; Thomas O'Dowd
Subject: RE: Query String Request Authentication(QSRA) support by S3 providers

Hi,

I have not seen the issue in extract template/iso after the changes. Works fine.

Thanks,
Sanjeev

-----Original Message-----
From: Min Chen [mailto:min.chen@citrix.com] 
Sent: Friday, July 19, 2013 5:08 AM
To: Jessica Wang
Cc: dev@cloudstack.apache.org; Edison Su; Thomas O'Dowd
Subject: Re: Query String Request Authentication(QSRA) support by S3 providers

Thanks Jessica. Tom, did you still see the issue?

-min

On 7/8/13 1:20 PM, "Jessica Wang" <Je...@citrix.com> wrote:

>Min,
>
>> would you please take a look at this to see if UI can disable 
>>decoding in displaying this download template url  returned from API?
>
>I just changed UI to not decode the URL returned in extractTemplate, 
>extractIso API.
>
>Jessica
>
>
>-----Original Message-----
>From: Min Chen
>Sent: Wednesday, July 03, 2013 5:53 PM
>To: dev@cloudstack.apache.org; Thomas O'Dowd
>Cc: Jessica Wang
>Subject: Re: Query String Request Authentication(QSRA) support by S3 
>providers
>
>Jessica, would you please take a look at this to see if UI can disable 
>decoding in displaying this download template url returned from API?
>
>Thanks
>-min
>
>On 7/3/13 5:38 PM, "Min Chen" <mi...@citrix.com> wrote:
>
>>By examining further what returned from extractTemplateCmd api, I 
>>realized that the URL returned from API is different from what is 
>>displayed from pop-up dialog from UI. Directly using the link returned 
>>from API (with / encoded as %2F can successfully download the 
>>template. So the issue may not be that bad to upgrade Amazon SDK, but 
>>a simple UI issue. That is, is it possible for UI not to decode 
>>URLEncoded string in this case?
>>
>>Thanks
>>-min
>>
>>On 7/3/13 5:22 PM, "Min Chen" <mi...@citrix.com> wrote:
>>
>>>Hi Tom,
>>>
>>>	I can reproduce this issue using Cloudian, after investigation, I 
>>>realized that this is a bug in Amazon SDK we have used, based on this
>>>thread: 
>>>http://stackoverflow.com/questions/15473582/amazon-s3-presigned-urls-
>>>esc
>>>a
>>>p
>>>e
>>>-the-slashes-in-the-key. When generatePresignedUrl is called it takes 
>>>the entire key and escapes it, and then creates a signature using the 
>>>escaped key. You cannot use the signature from the escaped key and 
>>>combine it with the unescaped key in the URL. See the bug code here:
>>>
>>>	String resourcePath = "/" +
>>>	((bucketName != null) ? bucketName + "/" : "") +
>>>	((key != null) ? ServiceUtils.urlEncode(key) : "") +
>>>	((subResource != null) ? "?" + subResource : "");
>>>
>>>We have two options to fix this:
>>>	1. Either upgrade Amazon SDK to use 1.4.3 version, someone in that 
>>>thread claimed that it is fixed in that version, but I haven't 
>>>checked that.
>>>Currently CloudStack is using 1.3.21. Not sure if this will break 
>>>CloudStack cloud_bridge.
>>>	2. Workaround by creating customized AmazonS3Client to change the 
>>>internal implementation on this.
>>>
>>>	Thanks
>>>	-min
>>>
>>>
>>>
>>>On 7/2/13 11:31 PM, "Thomas O'Dowd" <tp...@cloudian.com> wrote:
>>>
>>>>Excellent. The link is there now. Thank you Min. I verified that bug 
>>>>and closed it.
>>>>
>>>>However - now that I can finally click the download link... I ran 
>>>>into the issue that the link doesn't work on AWS or Cloudian. Please 
>>>>see this bug for details (latest 4.2 updates included in my test).
>>>>
>>>>    https://issues.apache.org/jira/browse/CLOUDSTACK-3341
>>>>
>>>>Tom.
>>>>
>>>>On Tue, 2013-07-02 at 22:54 +0000, Min Chen wrote:
>>>>> Hi Tom,
>>>>> 	I investigated this issue through the db dump you provided in the 
>>>>>bug,  this is an issue with our db view template_view creation 
>>>>>script, and it  has been fixed in resolving  another bug 
>>>>>(https://issues.apache.org/jira/browse/CLOUDSTACK-3314).
>>>>>I
>>>>> have verified the fix using your db dump on my local setup. Please 
>>>>>check  out latest 4.2 or master code to try again.
>>>>> 
>>>>> 	Thanks
>>>>> 	-min
>>>>> 
>>>>> On 7/2/13 2:18 PM, "Min Chen" <mi...@citrix.com> wrote:
>>>>> 
>>>>> >Tom, this seems like an issue with entry stored in our DB. I will
>>>>>take
>>>>>a
>>>>> >look at this bug and update you. Just to clarify, this symptom 
>>>>> >only happens when you register these templates to Amazon S3, not 
>>>>> >for
>>>>>Cloudian
>>>>> >or RiakCS S3, right?
>>>>> >
>>>>> >Thanks
>>>>> >-min
>>>>> >
>>>>> >On 7/1/13 7:27 PM, "Thomas O'Dowd" <tp...@cloudian.com> wrote:
>>>>> >
>>>>> >>Yes thanks Jessica. I re-opened the bug again. I know its not a 
>>>>> >>gui problem per-say in that the template is not ready to show 
>>>>> >>the
>>>>>download
>>>>> >>link. However, it never becomes ready is the actual problem. 
>>>>> >>What
>>>>>sets
>>>>> >>the "isready" property to true? As far as I can see, the objects 
>>>>> >>in
>>>>>the
>>>>> >>S3 stores (AWS or Cloudian) are complete and from my perspective
>>>>>"ready"
>>>>> >>to download/use. It sounds like a bug when registering the
>>>>>template.
>>>>> >>
>>>>> >>Tom.
>>>>> >>
>>>>> >>On Mon, 2013-07-01 at 18:54 +0000, Jessica Wang wrote:
>>>>> >>> Thomas,
>>>>> >>> 
>>>>> >>> I checked the data you provided.
>>>>> >>> 
>>>>> >>> The reason that the 2 templates("MyTiny", "AnotherTiny") have 
>>>>> >>>no download button is because they are not ready  (i.e. their 
>>>>> >>>"isready" property is false).
>>>>> >>> 
>>>>> >>> Download button is only available when "isready" property is
>>>>>true.
>>>>> >>> 
>>>>> >>> Jessica
>>>>> >>> 
>>>>> >>> -----Original Message-----
>>>>> >>> From: Thomas O'Dowd [mailto:tpodowd@cloudian.com]
>>>>> >>> Sent: Thursday, June 27, 2013 8:04 PM
>>>>> >>> To: Min Chen
>>>>> >>> Cc: dev@cloudstack.apache.org; Jessica Wang
>>>>> >>> Subject: Re: Query String Request Authentication(QSRA) support 
>>>>> >>> by
>>>>>S3
>>>>> >>>providers
>>>>> >>> 
>>>>> >>> Hi Min/Jessica,
>>>>> >>> 
>>>>> >>> I attached an image to that issue to show what what my browser 
>>>>> >>> is showing.
>>>>> >>> 
>>>>> >>>     https://issues.apache.org/jira/browse/CLOUDSTACK-3220
>>>>> >>> 
>>>>> >>> Tom.
>>>>> >>> 
>>>>> >>> On Fri, 2013-06-28 at 09:45 +0900, Thomas O'Dowd wrote:
>>>>> >>> > Hi Min,
>>>>> >>> > 
>>>>> >>> > Yes. I'll try it again today to check again but when I added
>>>>>Amazon
>>>>> >>>S3
>>>>> >>> > as the S3 secondary storage and uploaded a template, I was 
>>>>> >>> > not
>>>>>shown
>>>>> >>>the
>>>>> >>> > "download template" link. However - for Cloudian S3, I am 
>>>>> >>> > shown
>>>>>it so
>>>>> >>> > I'm wondering why.
>>>>> >>> > 
>>>>> >>> > Tom.
>>>>> >>> > 
>>>>> >>> > On Fri, 2013-06-28 at 00:26 +0000, Min Chen wrote:
>>>>> >>> > > Hi Tom,
>>>>> >>> > > 
>>>>> >>> > > 	Are you saying that you cannot see a Download Template
>>>>>button
>>>>>from
>>>>> >>>UI
>>>>> >>> > > when Amazon S3 is added as secondary storage? I only 
>>>>> >>> > > tested
>>>>>with
>>>>> >>>RiakCS
>>>>> >>> > > and Cloudian, so didn't see this issue. But I am CC 
>>>>> >>> > > Jessica
>>>>>her
>>>>>to
>>>>> >>>confirm
>>>>> >>> > > what special handling is done in UI to enable/disable a
>>>>>button
>>>>>from
>>>>> >>>UI.
>>>>> >>> > > 
>>>>> >>> > > 	Thanks
>>>>> >>> > > 	-min
>>>>> >>> > > 
>>>>> >>> > > On 6/27/13 5:23 PM, "Thomas O'Dowd" <tp...@cloudian.com>
>>>>>wrote:
>>>>> >>> > > 
>>>>> >>> > > >Hi Min,
>>>>> >>> > > >
>>>>> >>> > > >Can you check this bug? I'm trying to test this feature 
>>>>> >>> > > >for
>>>>>Amazon
>>>>> >>>but
>>>>> >>> > > >having no luck getting the Download template link/button 
>>>>> >>> > > >to
>>>>> >>>appear.
>>>>> >>> > > >
>>>>> >>> > > >https://issues.apache.org/jira/browse/CLOUDSTACK-3220
>>>>> >>> > > >
>>>>> >>> > > >Thanks,
>>>>> >>> > > >
>>>>> >>> > > >Tom.
>>>>> >>> > > >
>>>>> >>> > > >On Fri, 2013-06-21 at 17:21 +0000, Min Chen wrote:
>>>>> >>> > > >> John,
>>>>> >>> > > >> 
>>>>> >>> > > >> 	For S3, the api call createEntityExtractUrl is done on
>>>>> >>>management
>>>>> >>> > > >>server
>>>>> >>> > > >> side; while for NFS secondary storage, if the
>>>>>implementation
>>>>>of
>>>>> >>> > > >> createEntityExtractUrl will involve some code be 
>>>>> >>> > > >> executed
>>>>>in
>>>>> >>>ssvm to
>>>>> >>> > > >>copy
>>>>> >>> > > >> template from the install location to a public 
>>>>> >>> > > >>accessible
>>>>>web
>>>>> >>>server
>>>>> >>> > > >> location.
>>>>> >>> > > >> 	I don't quite understand some of your comments below.
>>>>>This
>>>>>API
>>>>> >>>is not
>>>>> >>> > > >> used to write any information to S3 bucket/directory. 
>>>>> >>> > > >> This
>>>>>is
>>>>> >>>used for
>>>>> >>> > > >> object already existed on S3, and we just provide a URL
>>>>>for
>>>>>user
>>>>> >>>to
>>>>> >>> > > >> download a template from S3, just like how Amazon 
>>>>> >>> > > >> provided
>>>>>user
>>>>> >>>a way to
>>>>> >>> > > >> user to extract a S3 object through generatePresignedUrl.
>>>>>We
>>>>>can
>>>>> >>>discuss
>>>>> >>> > > >> more on this on collaboration conference.
>>>>> >>> > > >> 
>>>>> >>> > > >> 	Thanks	
>>>>> >>> > > >> 	-min
>>>>> >>> > > >> 
>>>>> >>> > > >> 
>>>>> >>> > > >> 
>>>>> >>> > > >> On 6/21/13 7:25 AM, "John Burwell" <jb...@basho.com>
>>>>>wrote:
>>>>> >>> > > >> 
>>>>> >>> > > >> >Min,
>>>>> >>> > > >> >
>>>>> >>> > > >> >(I apologize for my belated reply -- I lost track of 
>>>>> >>> > > >> >this
>>>>>draft
>>>>> >>>in the
>>>>> >>> > > >> >chaos of the last couple of days.)
>>>>> >>> > > >> >
>>>>> >>> > > >> >Upon further review, I think I feel into the confusion
>>>>>between
>>>>> >>> > > >>management
>>>>> >>> > > >> >server and ssvm.  This code is executing on the
>>>>>management
>>>>> >>>server side,
>>>>> >>> > > >> >correct?  Based on my "corrected" understanding is
>>>>>correct,
>>>>>I
>>>>> >>>would
>>>>> >>> > > >>like
>>>>> >>> > > >> >to amend my thoughts.  Namely, I would like to see the
>>>>>driver
>>>>> >>> > > >>operations
>>>>> >>> > > >> >pushed out to the SSVM where we can use the stream.  
>>>>> >>> > > >> >As I
>>>>>think
>>>>> >>>about
>>>>> >>> > > >>it,
>>>>> >>> > > >> >the management server should not need to interact with
>>>>>the
>>>>> >>>driver.
>>>>> >>> > > >> >Simply yard up the DataStore attributes + details map 
>>>>> >>> > > >> >and
>>>>>other
>>>>> >>>extract
>>>>> >>> > > >> >parameters, and send them to the SSVM.  Using this
>>>>>information,
>>>>> >>>the S3
>>>>> >>> > > >> >driver could open a stream to write the template out 
>>>>> >>> > > >> >to
>>>>>the
>>>>> >>> > > >> >bucket/directory.  I recognize it changes the protocol
>>>>>between
>>>>> >>>the
>>>>> >>> > > >> >management server and SSVM, but it simply both sides 
>>>>> >>> > > >> >of
>>>>>the
>>>>> >>>operation
>>>>> >>> > > >>by
>>>>> >>> > > >> >allowing the DataStore information to be treated 
>>>>> >>> > > >> >opaquely
>>>>>until
>>>>> >>>it is
>>>>> >>> > > >> >consumed by the driver to execute the write operation.  
>>>>> >>> > > >> >I
>>>>>also
>>>>> >>> > > >>recognize
>>>>> >>> > > >> >that we may a little late in the cycle to address it 
>>>>> >>> > > >> >for
>>>>>4.2,
>>>>> >>>and it
>>>>> >>> > > >>may
>>>>> >>> > > >> >need to be part of the 4.3 enhancements.
>>>>> >>> > > >> >
>>>>> >>> > > >> >Thanks,
>>>>> >>> > > >> >-John
>>>>> >>> > > >> >
>>>>> >>> > > >> >On Jun 18, 2013, at 3:55 PM, Min Chen
>>>>><mi...@citrix.com>
>>>>> >>>wrote:
>>>>> >>> > > >> >
>>>>> >>> > > >> >> John,
>>>>> >>> > > >> >> 	In that case, how do we keep backward compatibility 
>>>>> >>> > > >> >> of
>>>>> >>> > > >>extractTemplate
>>>>> >>> > > >> >> api, which requires a URL in the response?
>>>>> >>> > > >> >>
>>>>> >>> > > >> >> 	Thanks
>>>>> >>> > > >> >> 	-min
>>>>> >>> > > >> >>
>>>>> >>> > > >> >> On 6/18/13 11:53 AM, "John Burwell"
>>>>><jb...@basho.com>
>>>>> >>>wrote:
>>>>> >>> > > >> >>
>>>>> >>> > > >> >>> Min,
>>>>> >>> > > >> >>>
>>>>> >>> > > >> >>> Looking through the code, I think we can simplify
>>>>>driver
>>>>> >>>operation
>>>>> >>> > > >>and
>>>>> >>> > > >> >>> increase robustness by changing
>>>>> >>> > > >> >>>ImageStoreDriver#createEntityExtractUrl()
>>>>> >>> > > >> >>> : String to ImageStoreDriver#readEntity(Š) :
>>>>>InputStream.
>>>>> >>>My first
>>>>> >>> > > >> >>> concern with the current implementation is that it
>>>>> >>>circumvents any
>>>>> >>> > > >> >>> connection pooling/resource management underlying
>>>>>client
>>>>> >>>libraries
>>>>> >>> > > >> >>> provide.  I/O streams provide a higher-level
>>>>>abstraction
>>>>> >>>that allows
>>>>> >>> > > >> >>> drivers to provide the orchestration components 
>>>>> >>> > > >> >>> with
>>>>>actual
>>>>> >>> > > >>resources
>>>>> >>> > > >> >>> rather String references.  Second, the current
>>>>>interface
>>>>> >>>seems to
>>>>> >>> > > >> >>>appears
>>>>> >>> > > >> >>> to assume that an http/https URL will be returned.
>>>>>With
>>>>>I/O
>>>>> >>> > > >>streams,
>>>>> >>> > > >> >>>we
>>>>> >>> > > >> >>> can support any client library capable of using the
>>>>>standard
>>>>> >>>I/O
>>>>> >>> > > >> >>> framework -- enabling us to support other protocols
>>>>>for
>>>>> >>>downloading
>>>>> >>> > > >> >>> templates in the future (e.g. RBD, local 
>>>>> >>> > > >> >>> filesystem,
>>>>>NBD,
>>>>> >>>etc).
>>>>> >>> > > >> >>>
>>>>> >>> > > >> >>> Thanks,
>>>>> >>> > > >> >>> -John
>>>>> >>> > > >> >>>
>>>>> >>> > > >> >>> On Jun 18, 2013, at 1:11 PM, Min Chen
>>>>><mi...@citrix.com>
>>>>> >>>wrote:
>>>>> >>> > > >> >>>
>>>>> >>> > > >> >>>> A new version of using generatePresignedUrl in
>>>>> >>> > > >>S3ImageStoreDriverImpl
>>>>> >>> > > >> >>>>is
>>>>> >>> > > >> >>>> checked into object_store.
>>>>> >>> > > >> >>>>
>>>>> >>> > > >> >>>> THanks
>>>>> >>> > > >> >>>> -min
>>>>> >>> > > >> >>>>
>>>>> >>> > > >> >>>> On 6/18/13 8:29 AM, "Min Chen" 
>>>>> >>> > > >> >>>> <mi...@citrix.com>
>>>>>wrote:
>>>>> >>> > > >> >>>>
>>>>> >>> > > >> >>>>> Yes, current code is in 
>>>>> >>> > > >> >>>>>S3ImageStoreDriverImpl.createEntityExtractUrl,
>>>>> >>> > > >> >>>>> which has a security issue mentioned in
>>>>>CLOUDSTACK-3030. I
>>>>> >>>am
>>>>> >>> > > >>going
>>>>> >>> > > >> >>>>>to
>>>>> >>> > > >> >>>>> change it to use generatePresignedUrl api from 
>>>>> >>> > > >> >>>>>AWS
>>>>>S3
>>>>>api.
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Thanks
>>>>> >>> > > >> >>>>> -min
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> From: John Burwell
>>>>> >>><jb...@basho.com>>
>>>>> >>> > > >> >>>>> Date: Tuesday, June 18, 2013 8:07 AM
>>>>> >>> > > >> >>>>> To: Min Chen
>>>>> >>><mi...@citrix.com>>
>>>>> >>> > > >> >>>>> Cc: Thomas O'Dowd 
>>>>> >>> > > >> >>>>><tp...@cloudian.com>
>>>>> >>> > > >> >>>>>>,
>>>>> >>> > > >> >>>>>
>>>>> >>>"dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
>>>>> >>> > > >> >>>>>
>>>>> >>><de...@cloudstack.apache.org>>
>>>>> >>> > > >> >>>>> Subject: Re: Query String Request
>>>>>Authentication(QSRA)
>>>>> >>>support by
>>>>> >>> > > >>S3
>>>>> >>> > > >> >>>>> providers
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Min,
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Is the code checked into the object_store branch?
>>>>>If
>>>>>so,
>>>>> >>>which
>>>>> >>> > > >>lines
>>>>> >>> > > >> >>>>> in
>>>>> >>> > > >> >>>>> S3TemplateDownloader?
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Thanks,
>>>>> >>> > > >> >>>>> -John
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> On Jun 18, 2013, at 12:39 AM, Min Chen 
>>>>> >>> > > >> >>>>> <mi...@citrix.com>>
>>>>>wrote:
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Hi John,
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> This is regarding extractTemplate api, where for
>>>>> >>>extractable
>>>>> >>> > > >> >>>>>template,
>>>>> >>> > > >> >>>>> users can click "Download Template" button from 
>>>>> >>> > > >> >>>>>UI
>>>>>to
>>>>>get
>>>>> >>>a http
>>>>> >>> > > >>url
>>>>> >>> > > >> >>>>>to
>>>>> >>> > > >> >>>>> download the template already stored at S3 
>>>>> >>> > > >> >>>>>without
>>>>> >>>providing S3
>>>>> >>> > > >> >>>>> credentials. In 4.1, we don't have this issue, 
>>>>> >>> > > >> >>>>> since
>>>>>the
>>>>> >>>URL
>>>>> >>> > > >>returned
>>>>> >>> > > >> >>>>> is
>>>>> >>> > > >> >>>>> the public web server location hosted in ssvm, 
>>>>> >>> > > >> >>>>> and
>>>>>in
>>>>>4.2,
>>>>> >>>we are
>>>>> >>> > > >> >>>>> returning URL pointing to s3 object. Without 
>>>>> >>> > > >> >>>>> setting
>>>>>ACL
>>>>> >>>to the S3
>>>>> >>> > > >> >>>>> object, user cannot directly click the URL 
>>>>> >>> > > >> >>>>> returned
>>>>>from
>>>>> >>> > > >> >>>>> extractTemplate
>>>>> >>> > > >> >>>>> api to download the template without providing
>>>>> >>>credentials. By
>>>>> >>> > > >> >>>>>reading
>>>>> >>> > > >> >>>>> the AWS SDK doc today, I ran across the following
>>>>>API
>>>>>that
>>>>> >>>I may
>>>>> >>> > > >>be
>>>>> >>> > > >> >>>>> able
>>>>> >>> > > >> >>>>> to use for this purpose:
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.
>>>>>>>>>>>>>>>htm
>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>?
>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>s-
>>>>> >>>>>>>>>>e
>>>>> >>>>>>>>>>xt
>>>>> >>> > > >>>>>>>er
>>>>> >>> > > >> >>>>>na
>>>>> >>> > > >> >>>>> l=
>>>>> >>> > > >> >>>>> true>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>generatePresignedUrl<http://docs.aws.amazon.com/AWSJavaSD
>>>>>>>>>>>>>>>K/l
>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>t
>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>st
>>>>> >>>>>>>>>>/
>>>>> >>>>>>>>>>ja
>>>>> >>> > > >>>>>>>va
>>>>> >>> > > >> >>>>>do
>>>>> >>> > > >> >>>>> c/
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>com/amazonaws/services/s3/AmazonS3Client.html#generatePre
>>>>>>>>>>>>>>>sig
>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>e
>>>>>>>>>>>>>>>d
>>>>>>>>>>>>>>>Ur
>>>>> >>>>>>>>>>l
>>>>> >>>>>>>>>>%2
>>>>> >>> > > >>>>>>>8j
>>>>> >>> > > >> >>>>>av
>>>>> >>> > > >> >>>>> a.
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.
>>>>>>>>>>>>>>>ama
>>>>>>>>>>>>>>>z
>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>aw
>>>>> >>>>>>>>>>s
>>>>> >>>>>>>>>>.H
>>>>> >>> > > >>>>>>>tt
>>>>> >>> > > >> >>>>>pM
>>>>> >>> > > >> >>>>> et
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/ja
>>>>>>>>>>>>>>>va/
>>>>>>>>>>>>>>>l
>>>>>>>>>>>>>>>a
>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>g/
>>>>> >>>>>>>>>>S
>>>>> >>>>>>>>>>tr
>>>>> >>> > > >>>>>>>in
>>>>> >>> > > >> >>>>>g.
>>>>> >>> > > >> >>>>> ht
>>>>> >>> > > >> >>>>> ml?is-external=true> bucketName,
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/
>>>>>>>>>>>>>>>Str
>>>>>>>>>>>>>>>i
>>>>>>>>>>>>>>>n
>>>>>>>>>>>>>>>g
>>>>>>>>>>>>>>>.h
>>>>> >>>>>>>>>>t
>>>>> >>>>>>>>>>ml
>>>>> >>> > > >>>>>>>?i
>>>>> >>> > > >> >>>>>s-
>>>>> >>> > > >> >>>>> ex
>>>>> >>> > > >> >>>>> ternal=true> key,
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.
>>>>>>>>>>>>>>>h
>>>>>>>>>>>>>>>t
>>>>>>>>>>>>>>>m
>>>>>>>>>>>>>>>l?
>>>>> >>>>>>>>>>i
>>>>> >>>>>>>>>>s-
>>>>> >>> > > >>>>>>>ex
>>>>> >>> > > >> >>>>>te
>>>>> >>> > > >> >>>>> rn
>>>>> >>> > > >> >>>>> al=true> expiration,
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> 
>>>>> >>> > > 
>>>>> 
>>>>>>>>>>>>>>>HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/latest/j
>>>>>>>>>>>>>>>ava
>>>>>>>>>>>>>>>d
>>>>>>>>>>>>>>>o
>>>>>>>>>>>>>>>c
>>>>>>>>>>>>>>>/c
>>>>> >>>>>>>>>>o
>>>>> >>>>>>>>>>m/
>>>>> >>> > > >>>>>>>am
>>>>> >>> > > >> >>>>>az
>>>>> >>> > > >> >>>>> on
>>>>> >>> > > >> >>>>> aws/HttpMethod.html> method)
>>>>> >>> > > >> >>>>>         Returns a pre-signed URL for accessing an
>>>>>Amazon
>>>>> >>>S3
>>>>> >>> > > >>resource.
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> This is along the same line as QSRA mentioned by
>>>>>Tom,
>>>>>by
>>>>> >>>wrapped
>>>>> >>> > > >>in
>>>>> >>> > > >> >>>>> AmazonS3Client for easy consumption. By using 
>>>>> >>> > > >> >>>>> this
>>>>>method,
>>>>> >>>I think
>>>>> >>> > > >> >>>>> that I
>>>>> >>> > > >> >>>>> don't need to change ACL of S3 object to open a
>>>>>security
>>>>> >>>hole.
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Thanks
>>>>> >>> > > >> >>>>> -min
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> From: John Burwell
>>>>> >>><jb...@basho.com>>
>>>>> >>> > > >> >>>>> Date: Monday, June 17, 2013 7:38 PM
>>>>> >>> > > >> >>>>> To: Min Chen
>>>>> >>><mi...@citrix.com>>
>>>>> >>> > > >> >>>>> Cc: Thomas O'Dowd 
>>>>> >>> > > >> >>>>><tp...@cloudian.com>
>>>>> >>> > > >> >>>>>>,
>>>>> >>> > > >> >>>>>
>>>>> >>>"dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
>>>>> >>> > > >> >>>>>
>>>>> >>><de...@cloudstack.apache.org>>
>>>>> >>> > > >> >>>>> Subject: Re: Query String Request
>>>>>Authentication(QSRA)
>>>>> >>>support by
>>>>> >>> > > >>S3
>>>>> >>> > > >> >>>>> providers
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Min,
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Why are we mucking with ACLs at all?  The best
>>>>>security
>>>>> >>>practice
>>>>> >>> > > >> >>>>>would
>>>>> >>> > > >> >>>>> be
>>>>> >>> > > >> >>>>> to create a bucket for CloudStack's use and 
>>>>> >>> > > >> >>>>>assign
>>>>>it
>>>>>a
>>>>> >>>dedicated
>>>>> >>> > > >> >>>>> access
>>>>> >>> > > >> >>>>> key and secret key pair with read/write access 
>>>>> >>> > > >> >>>>> only
>>>>>to
>>>>> >>>that
>>>>> >>> > > >>bucket.
>>>>> >>> > > >> >>>>> Requiring an administrative account to an object
>>>>>store
>>>>> >>>opens an
>>>>> >>> > > >> >>>>> unnecessarily large attack surface.  Therefore, 
>>>>> >>> > > >> >>>>> as
>>>>> >>>implemented in
>>>>> >>> > > >> >>>>>4.1,
>>>>> >>> > > >> >>>>> we
>>>>> >>> > > >> >>>>> should defer bucket creation, ACL assignment, and
>>>>> >>>credential
>>>>> >>> > > >>creation
>>>>> >>> > > >> >>>>> to
>>>>> >>> > > >> >>>>> the administrator/operator.
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Thanks,
>>>>> >>> > > >> >>>>> -John
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> On Jun 17, 2013, at 1:15 PM, Min Chen 
>>>>> >>> > > >> >>>>> <mi...@citrix.com>>
>>>>>wrote:
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Tom filed a very good bug for ACL setting change 
>>>>> >>> > > >> >>>>> on
>>>>>S3
>>>>> >>>object when
>>>>> >>> > > >> >>>>> users
>>>>> >>> > > >> >>>>> issue extractTemplate API
>>>>> >>> > > >> >>>>>
>>>>>(https://issues.apache.org/jira/browse/CLOUDSTACK-3030),
>>>>> >>>and his
>>>>> >>> > > >> >>>>> recommendation of using Query String Request
>>>>> >>>Authentication (QSRA)
>>>>> >>> > > >> >>>>> alternative sounds like a right approach to fix 
>>>>> >>> > > >> >>>>> this
>>>>>bug.
>>>>> >>>Before
>>>>> >>> > > >> >>>>> implementing it, I would like to confirm if QSRA
>>>>>should be
>>>>> >>> > > >>supported
>>>>> >>> > > >> >>>>>by
>>>>> >>> > > >> >>>>> all S3 providers if they claim that they are AWS 
>>>>> >>> > > >> >>>>>s3
>>>>> >>>compatible. If
>>>>> >>> > > >> >>>>>so,
>>>>> >>> > > >> >>>>> we
>>>>> >>> > > >> >>>>> will make this assumption in our code. Based on 
>>>>> >>> > > >> >>>>>Tom,
>>>>> >>>Cloudian is
>>>>> >>> > > >> >>>>> supporting it. How about RiakCS, John?
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>> Thanks
>>>>> >>> > > >> >>>>> -min
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>>
>>>>> >>> > > >> >>>>
>>>>> >>> > > >> >>>
>>>>> >>> > > >> >>
>>>>> >>> > > >> >
>>>>> >>> > > >> 
>>>>> >>> > > >
>>>>> >>> > > >--
>>>>> >>> > > >Cloudian KK - http://www.cloudian.com/get-started.html
>>>>> >>> > > >Fancy 100TB of full featured S3 Storage?
>>>>> >>> > > >Checkout the Cloudian(R) Community Edition!
>>>>> >>> > > >
>>>>> >>> > > 
>>>>> >>> > 
>>>>> >>> 
>>>>> >>
>>>>> >>--
>>>>> >>Cloudian KK - http://www.cloudian.com/get-started.html
>>>>> >>Fancy 100TB of full featured S3 Storage?
>>>>> >>Checkout the Cloudian(R) Community Edition!
>>>>> >>
>>>>> >
>>>>> 
>>>>
>>>>--
>>>>Cloudian KK - http://www.cloudian.com/get-started.html
>>>>Fancy 100TB of full featured S3 Storage?
>>>>Checkout the Cloudian(R) Community Edition!
>>>>
>>>
>>
>