You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jclouds.apache.org by Andrew Gaul <ga...@apache.org> on 2017/05/05 21:36:07 UTC
Re: Async API implementation for storage
I demonstrate how to use the existing InputStream getBlob interface
asynchronously here:
https://github.com/andrewgaul/s3proxy/pull/220
Callers can interact with putBlob in a similar asynchronous manner if we
add an OutputStream variant to jclouds:
https://issues.apache.org/jira/browse/JCLOUDS-769
HTTP drivers generally provide OutputStream support but our HTTP
interface does not expose it; see BaseHttpCommandExecutorService.invoke
and JavaUrlHttpCommandExecutorService.writePayloadToConnection.
Providing this interface would fix a couple of issues in jclouds,
including aborting operations:
https://issues.apache.org/jira/browse/JCLOUDS-627
Allowing the caller to drive IO enables more esoteric use cases like
writing to two object stores simultaneously:
https://github.com/andrewgaul/s3proxy/issues/117
I strongly prefer to add a generally useful OutputStream interface which
solves the asynchronous use case instead of a special-purpose
asynchronous interface which cannot be used for other things.
On Fri, Apr 28, 2017 at 10:30:21AM -0700, Andrew Gaul wrote:
> Kishore, please address the comments about the OutputStream approach I
> made a month ago and again last week:
>
> https://lists.apache.org/thread.html/d78fe3db757ea4430a92a8af31e1578e776898488a67844f03b7f356@%3Cdev.jclouds.apache.org%3E
> https://lists.apache.org/thread.html/1062debf3e4253bbc46485672051a1bc8adaa0008e35da0bf514e495@%3Cdev.jclouds.apache.org%3E
>
> While this approach is simpler it is generally more useful to users. I
> will not review implementations that have not completed a design review.
>
> On Fri, Apr 28, 2017 at 10:47:27AM -0000, kishore kumar wrote:
> > Hi,
> >
> > I have completed adding putBlobAsync method. I have done all the changes specified by the community.
> > Here it the pull request for that.
> >
> > https://github.com/SpandanThakur/jclouds/pull/2
> >
> > Some community members suggested having new interface for asynchronous calls. I have done it as a separate pull request.
> > https://github.com/SpandanThakur/jclouds/pull/4
> >
> > Please let me know your comments and next steps to take it to master branch
>
> --
> Andrew Gaul
> http://gaul.org/
--
Andrew Gaul
http://gaul.org/
Re: Async API implementation for storage
Posted by kishore kumar <ki...@gmail.com>.
Hi Andrew,
We are compiling the performance numbers. I will post the perf numbers in a day or two.
On 2017-05-12 06:14 (+0530), Andrew Gaul <ga...@apache.org> wrote:
> I am happy that you resolved your email issues. You commented on my
> partially async design but you have not shown that your fully async
> design is actually faster. Please provide benchmarks as requested here:
>
> https://github.com/andrewgaul/s3proxy/pull/220#discussion_r116132467
>
> Adding a new HTTP code path has a higher barrier for inclusion than a
> simple bug fix. The jclouds team might need to maintain this code for a
> long time so the burden of proof is on you to show that we need it.
>
> On Fri, May 12, 2017 at 12:20:05AM -0000, kishore kumar wrote:
> > Hi Andrew,
> >
> > I have reviewed your pull request and the approach you specified doesn't solve what we are trying to address. I replied to the pull request.
>
> --
> Andrew Gaul
> http://gaul.org/
>
Re: Async API implementation for storage
Posted by Andrew Gaul <ga...@apache.org>.
I am happy that you resolved your email issues. You commented on my
partially async design but you have not shown that your fully async
design is actually faster. Please provide benchmarks as requested here:
https://github.com/andrewgaul/s3proxy/pull/220#discussion_r116132467
Adding a new HTTP code path has a higher barrier for inclusion than a
simple bug fix. The jclouds team might need to maintain this code for a
long time so the burden of proof is on you to show that we need it.
On Fri, May 12, 2017 at 12:20:05AM -0000, kishore kumar wrote:
> Hi Andrew,
>
> I have reviewed your pull request and the approach you specified doesn't solve what we are trying to address. I replied to the pull request.
--
Andrew Gaul
http://gaul.org/
Re: Async API implementation for storage
Posted by kishore kumar <ki...@gmail.com>.
Hi Andrew,
I have reviewed your pull request and the approach you specified doesn't solve what we are trying to address. I replied to the pull request.
Re: Async API implementation for storage
Posted by kishore kumar <ki...@gmail.com>.
Hi Andrew,
What is the process for design review. Can you let me know the steps involved in it?
On 2017-05-05 14:36 (-0700), Andrew Gaul <ga...@apache.org> wrote:
> I demonstrate how to use the existing InputStream getBlob interface
> asynchronously here:
>
> https://github.com/andrewgaul/s3proxy/pull/220
>
> Callers can interact with putBlob in a similar asynchronous manner if we
> add an OutputStream variant to jclouds:
>
> https://issues.apache.org/jira/browse/JCLOUDS-769
>
> HTTP drivers generally provide OutputStream support but our HTTP
> interface does not expose it; see BaseHttpCommandExecutorService.invoke
> and JavaUrlHttpCommandExecutorService.writePayloadToConnection.
> Providing this interface would fix a couple of issues in jclouds,
> including aborting operations:
>
> https://issues.apache.org/jira/browse/JCLOUDS-627
>
> Allowing the caller to drive IO enables more esoteric use cases like
> writing to two object stores simultaneously:
>
> https://github.com/andrewgaul/s3proxy/issues/117
>
> I strongly prefer to add a generally useful OutputStream interface which
> solves the asynchronous use case instead of a special-purpose
> asynchronous interface which cannot be used for other things.
>
> On Fri, Apr 28, 2017 at 10:30:21AM -0700, Andrew Gaul wrote:
> > Kishore, please address the comments about the OutputStream approach I
> > made a month ago and again last week:
> >
> > https://lists.apache.org/thread.html/d78fe3db757ea4430a92a8af31e1578e776898488a67844f03b7f356@%3Cdev.jclouds.apache.org%3E
> > https://lists.apache.org/thread.html/1062debf3e4253bbc46485672051a1bc8adaa0008e35da0bf514e495@%3Cdev.jclouds.apache.org%3E
> >
> > While this approach is simpler it is generally more useful to users. I
> > will not review implementations that have not completed a design review.
> >
> > On Fri, Apr 28, 2017 at 10:47:27AM -0000, kishore kumar wrote:
> > > Hi,
> > >
> > > I have completed adding putBlobAsync method. I have done all the changes specified by the community.
> > > Here it the pull request for that.
> > >
> > > https://github.com/SpandanThakur/jclouds/pull/2
> > >
> > > Some community members suggested having new interface for asynchronous calls. I have done it as a separate pull request.
> > > https://github.com/SpandanThakur/jclouds/pull/4
> > >
> > > Please let me know your comments and next steps to take it to master branch
> >
> > --
> > Andrew Gaul
> > http://gaul.org/
>
> --
> Andrew Gaul
> http://gaul.org/
>