aws s3 list objects v2

[Justinus Menzel <ju...@vyasa.com>]

Hi,

I've been playing around with s3proxy and jclouds and was wondering if
there are any plans to support s3's list objects v2 call.
https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
Seems like some tools already rely on V2, for example the aws CLI command.

Best,

-Justinus

Re: aws s3 list objects v2

[Andrew Gaul <ga...@apache.org>]

Great, let's continue the discussion at
https://github.com/jclouds/jclouds/pull/1267 .

jclouds commits must merge to master first although I can backport them
to 2.1.x.  Timing releases is a sore point but S3Proxy already upgraded
to 2.1.2-SNAPSHOT so S3Proxy's snapshot releases and Docker images will
immediately include such fixes.  Unfortunately its next release will now
be gated on jclouds.

On Tue, Jan 15, 2019 at 10:20:45AM -0500, Justinus Menzel wrote:
> Andrew, thank you for responding.
> I really need both parts. Hence I made changes to s3proxy which rely on
> changes to jclouds.
> I can setup the pull request with target https://github.com/gaul/s3proxy
> But it also needs the jclouds change where jclouds back-end issues the
> 'list-type=2' request.
> I'll try to setup a pull request into the jclouds repo.
> Not sure how easily this PR will get accepted and merged. Apparently
> changes need to go into master.
> How long before master becomes stable and can be used for building/
> releasing s3proxy?
> Anyways, I'll give it a shot.
> Best,

-- 
Andrew Gaul
http://gaul.org/

Re: aws s3 list objects v2

[Justinus Menzel <ju...@vyasa.com>]

Andrew, thank you for responding.
I really need both parts. Hence I made changes to s3proxy which rely on
changes to jclouds.
I can setup the pull request with target https://github.com/gaul/s3proxy
But it also needs the jclouds change where jclouds back-end issues the
'list-type=2' request.
I'll try to setup a pull request into the jclouds repo.
Not sure how easily this PR will get accepted and merged. Apparently
changes need to go into master.
How long before master becomes stable and can be used for building/
releasing s3proxy?
Anyways, I'll give it a shot.
Best,
-Justinus



On Thu, Jan 3, 2019 at 10:01 PM Andrew Gaul <ga...@apache.org> wrote:

> On Fri, Nov 30, 2018 at 09:53:31AM -0500, Justinus Menzel wrote:
> > I've been playing around with s3proxy and jclouds and was wondering if
> > there are any plans to support s3's list objects v2 call.
> > https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
> > Seems like some tools already rely on V2, for example the aws CLI
> command.
>
> There are two parts to this; the S3Proxy front-end interpretation of
> `continuation-token` and the jclouds back-end issuing of `list-type=2`.
> The latter is easy to implement as an additional call in `S3Client`;
> would you like to submit a pull request?  I suspect you really want the
> former support, however.  Honestly I have read the documentation a few
> times and do not understand the subtlety between v1 and v2 calls; these
> look the same?
>
> It looks like you implemented this but sent the pull request to the
> wrong repository:
>
> https://github.com/vyasa-analytics/s3proxy/pull/1
>
> Could you send this to https://github.com/gaul/s3proxy ?
>
> It is unfortunate that clients do not fail back from v2 to v1
> automatically like some do for v4 to v2 authentication, but S3Proxy
> should just add support for v2.
>
> --
> Andrew Gaul
> http://gaul.org/
>

Re: aws s3 list objects v2

[Andrew Gaul <ga...@apache.org>]

On Fri, Nov 30, 2018 at 09:53:31AM -0500, Justinus Menzel wrote:
> I've been playing around with s3proxy and jclouds and was wondering if
> there are any plans to support s3's list objects v2 call.
> https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
> Seems like some tools already rely on V2, for example the aws CLI command.

There are two parts to this; the S3Proxy front-end interpretation of
`continuation-token` and the jclouds back-end issuing of `list-type=2`.
The latter is easy to implement as an additional call in `S3Client`;
would you like to submit a pull request?  I suspect you really want the
former support, however.  Honestly I have read the documentation a few
times and do not understand the subtlety between v1 and v2 calls; these
look the same?

It looks like you implemented this but sent the pull request to the
wrong repository:

https://github.com/vyasa-analytics/s3proxy/pull/1

Could you send this to https://github.com/gaul/s3proxy ?

It is unfortunate that clients do not fail back from v2 to v1
automatically like some do for v4 to v2 authentication, but S3Proxy
should just add support for v2.

-- 
Andrew Gaul
http://gaul.org/