You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jclouds.apache.org by Archana C <ca...@yahoo.co.uk> on 2017/02/03 05:22:15 UTC
jClouds 2.0 MultiPart Upload
Hi
I have written a sample code for multipart upload using jClouds-2.0
Properties overrides = new Properties();
BlobStoreContext context = ContextBuilder.newBuilder("openstack-swift")
.endpoint("http://x.xxx.xx.xx:5000/v2.0")
.credentials("xx:xx", "xx")
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
blobStore.createContainerInLocation(null, CONTAINER_NAME);
Path path = Paths.get("test2");
File f = new File("test2");
byte []byteArray = Files.readAllBytes(path);
Payload payload = newByteSourcePayload(wrap(byteArray));
PutOptions opt = new PutOptions();
opt.multipart();
Blob blob = blobStore.blobBuilder(OBJECT_NAME)
.payload(payload).contentLength(f.length())
.build();
String etag = blobStore.putBlob(CONTAINER_NAME, blob, opt);
test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
10:21:52.355 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09d4
10:21:53.129 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving response 1344471693: HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41-0058940c68
10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
10:21:53.373 [user thread 1] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking object:put
10:21:53.373 [user thread 0] DEBUG o.j.rest.internal.InvokeHttpMethod - >> invoking object:put
10:21:53.374 [user thread 1] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/1486097513.327000/0/33554432/00000001 HTTP/1.1
10:21:53.376 [user thread 0] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/1486097513.327000/0/33554432/00000000 HTTP/1.1
10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal.HttpWire - over limit 3145728/262144: wrote temp file
10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal.HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1380)
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1373)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.java:650)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.java:628)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putBlob(RegionScopedSwiftBlobStore.java:274)
at jclouds20.App.main(App.java:83)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuilder.ensureCapacityImpl(StringBuilder.java:342)
at java.lang.StringBuilder.append(StringBuilder.java:208)
at org.jclouds.logging.internal.Wire.wire(Wire.java:68)
at org.jclouds.logging.internal.Wire.copy(Wire.java:99)
at org.jclouds.logging.internal.Wire.output(Wire.java:176)
at org.jclouds.logging.internal.Wire.output(Wire.java:143)
at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:296)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy68.put(Unknown Source)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.uploadMultipartPart(RegionScopedSwiftBlobStore.java:529)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.java:666)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.java:653)
at java.util.concurrent.FutureTask.run(FutureTask.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread.java:798)
at java.lang.Thread.getStackTraceImpl(Native Method)
at java.lang.Thread.getStackTrace(Thread.java:1133)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.java:647)
... 3 more
Feb 03, 2017 10:22:03 AM com.google.common.util.concurrent.Futures$CombinedFuture setExceptionAndMaybeLog
SEVERE: input future failed.
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuilder.ensureCapacityImpl(StringBuilder.java:342)
at java.lang.StringBuilder.append(StringBuilder.java:208)
at org.jclouds.logging.internal.Wire.wire(Wire.java:68)
at org.jclouds.logging.internal.Wire.copy(Wire.java:99)
at org.jclouds.logging.internal.Wire.output(Wire.java:176)
at org.jclouds.logging.internal.Wire.output(Wire.java:143)
at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:296)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy68.put(Unknown Source)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.uploadMultipartPart(RegionScopedSwiftBlobStore.java:529)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.java:666)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.java:653)
at java.util.concurrent.FutureTask.run(FutureTask.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread.java:798)
at java.lang.Thread.getStackTraceImpl(Native Method)
at java.lang.Thread.getStackTrace(Thread.java:1133)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.java:647)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.java:628)
at org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putBlob(RegionScopedSwiftBlobStore.java:274)
at jclouds20.App.main(App.java:83)
Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
RegardsArchana
Re: jClouds 2.0 MultiPart Upload
Posted by Zack Shoylev <za...@RACKSPACE.COM>.
It will use an internal executorservice, if i remember correctly. I don't remember how it was configured, though, I'll have to look. Probably slow if not even sequential.
-Zack
________________________________
From: Archana C <ca...@yahoo.co.uk>
Sent: Saturday, February 4, 2017 12:37 AM
To: user@jclouds.apache.org
Subject: Re: jClouds 2.0 MultiPart Upload
Does it mean if we are not specifying executorservice, the upload happens in sequential ?
Regards
Archana
On Saturday, 4 February 2017, 10:34, Andrew Gaul <ga...@apache.org> wrote:
PutOptions *takes* an ExecutorService which allows multiple threads to
concurrently upload multiple parts.
On Sat, Feb 04, 2017 at 03:34:30AM +0000, Archana C wrote:
> Hi
>
> I think the question was not clear. Parallel upload of multiple file is fine and that can be achieved by using executorservice.
> The question here is, does multipartUpload i.e uploading of each part is happening in parallel ?
> Does sequential upload of part deprecated ?
> RegardsArchana
>
>
>
> On Saturday, 4 February 2017, 1:30, Andrew Gaul <ga...@apache.org>> wrote:
>
>
> We rewrote multi-part uploads in jclouds 2.0. You should pass an
> ExecutorService and via PutOptions in your call to BlobStore.putBlob.
>
> On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote:
> > Hi
> >
> > Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart uploads are parallel now ?
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 18:39, Archana C <ca...@yahoo.co.uk>> wrote:
> >
> >
> > Thanks it helped
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 12:06, Ignasi Barrera <na...@apache.org>> wrote:
> >
> >
> > It looks like the OOM exception is thrown when writing the wire logs. When using the blob store apis you might see binary data in the logs, as the "jclouds.wire" logger logs the response/request payloads which might be huge for some blobs and can cause this kind of exceptions.
> > Could you try disabling the wire logs? (I recommend doing this for production environments).
> > Perhaps for your use case the "jclouds.headers" are enough; that will log all request/reponse path and headers but skip the bodies.
> > More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
> > https://issues.apache.org/jira/browse/JCLOUDS-932
> >
> >
> > HTH!
> > I.
> > On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk>> wrote:
> >
> > Hi
> >
> > I have written a sample code for multipart upload using jClouds-2.0
> > Properties overrides = new Properties();
> > BlobStoreContext context = ContextBuilder.newBuilder(" openstack-swift")
> > .endpoint("http://x.xxx.xx.xx: <http://x.xxx.xx.xx/> 5000/v2.0")
> > .credentials("xx:xx", "xx")
> > .overrides(overrides)
> > .modules(modules)
> > .buildView(BlobStoreContext. class);
> > BlobStore blobStore = context.getBlobStore();
> > blobStore. createContainerInLocation( null, CONTAINER_NAME);
> > Path path = Paths.get("test2");
> > File f = new File("test2");
> > byte []byteArray = Files.readAllBytes(path);
> > Payload payload = newByteSourcePayload(wrap( byteArray));
> > PutOptions opt = new PutOptions();
> > opt.multipart();
> > Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
> > .payload(payload). contentLength(f.length())
> > .build();
> > String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
> > test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
> > 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> > 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09 d4
> > 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Receiving response 1344471693: HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
> > 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41- 0058940c68
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
> > 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> > 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> > 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
> > 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
> > 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over limit 3145728/262144: wrote temp file
> > 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util. concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
> > at com.google.common.util. concurrent.Futures. wrapAndThrowUnchecked(Futures. java:1380)
> > at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: 1373)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:650)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> > ... 3 more
> > Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ CombinedFuture setExceptionAndMaybeLog
> > SEVERE: input future failed.
> > java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> >
> > Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
> > Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
> >
> > RegardsArchana
> >
> >
> >
> >
> >
> >
> >
> >
>
> --
> Andrew Gaul
> http://gaul.org/
>
>
>
--
Andrew Gaul
http://gaul.org/
Re: jClouds 2.0 MultiPart Upload
Posted by Archana C <ca...@yahoo.co.uk>.
Does it mean if we are not specifying executorservice, the upload happens in sequential ?
Regards
Archana
On Saturday, 4 February 2017, 10:34, Andrew Gaul <ga...@apache.org> wrote:
PutOptions *takes* an ExecutorService which allows multiple threads to
concurrently upload multiple parts.
On Sat, Feb 04, 2017 at 03:34:30AM +0000, Archana C wrote:
> Hi
>
> I think the question was not clear. Parallel upload of multiple file is fine and that can be achieved by using executorservice.
> The question here is, does multipartUpload i.e uploading of each part is happening in parallel ?
> Does sequential upload of part deprecated ?
> RegardsArchana
>
>
>
> On Saturday, 4 February 2017, 1:30, Andrew Gaul <ga...@apache.org> wrote:
>
>
> We rewrote multi-part uploads in jclouds 2.0. You should pass an
> ExecutorService and via PutOptions in your call to BlobStore.putBlob.
>
> On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote:
> > Hi
> >
> > Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart uploads are parallel now ?
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 18:39, Archana C <ca...@yahoo.co.uk> wrote:
> >
> >
> > Thanks it helped
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 12:06, Ignasi Barrera <na...@apache.org> wrote:
> >
> >
> > It looks like the OOM exception is thrown when writing the wire logs. When using the blob store apis you might see binary data in the logs, as the "jclouds.wire" logger logs the response/request payloads which might be huge for some blobs and can cause this kind of exceptions.
> > Could you try disabling the wire logs? (I recommend doing this for production environments).
> > Perhaps for your use case the "jclouds.headers" are enough; that will log all request/reponse path and headers but skip the bodies.
> > More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
> > https://issues.apache.org/jira/browse/JCLOUDS-932
> >
> >
> > HTH!
> > I.
> > On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk> wrote:
> >
> > Hi
> >
> > I have written a sample code for multipart upload using jClouds-2.0
> > Properties overrides = new Properties();
> > BlobStoreContext context = ContextBuilder.newBuilder(" openstack-swift")
> > .endpoint("http://x.xxx.xx.xx: 5000/v2.0")
> > .credentials("xx:xx", "xx")
> > .overrides(overrides)
> > .modules(modules)
> > .buildView(BlobStoreContext. class);
> > BlobStore blobStore = context.getBlobStore();
> > blobStore. createContainerInLocation( null, CONTAINER_NAME);
> > Path path = Paths.get("test2");
> > File f = new File("test2");
> > byte []byteArray = Files.readAllBytes(path);
> > Payload payload = newByteSourcePayload(wrap( byteArray));
> > PutOptions opt = new PutOptions();
> > opt.multipart();
> > Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
> > .payload(payload). contentLength(f.length())
> > .build();
> > String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
> > test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
> > 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> > 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09 d4
> > 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Receiving response 1344471693: HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
> > 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41- 0058940c68
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
> > 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> > 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> > 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
> > 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
> > 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over limit 3145728/262144: wrote temp file
> > 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util. concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
> > at com.google.common.util. concurrent.Futures. wrapAndThrowUnchecked(Futures. java:1380)
> > at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: 1373)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:650)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> > ... 3 more
> > Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ CombinedFuture setExceptionAndMaybeLog
> > SEVERE: input future failed.
> > java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> >
> > Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
> > Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
> >
> > RegardsArchana
> >
> >
> >
> >
> >
> >
> >
> >
>
> --
> Andrew Gaul
> http://gaul.org/
>
>
>
--
Andrew Gaul
http://gaul.org/
Re: jClouds 2.0 MultiPart Upload
Posted by Andrew Gaul <ga...@apache.org>.
PutOptions *takes* an ExecutorService which allows multiple threads to
concurrently upload multiple parts.
On Sat, Feb 04, 2017 at 03:34:30AM +0000, Archana C wrote:
> Hi
>
> I think the question was not clear. Parallel upload of multiple file is fine and that can be achieved by using executorservice.
> The question here is, does multipartUpload i.e uploading of each part is happening in parallel ?
> Does sequential upload of part deprecated ?
> RegardsArchana
>
>
>
> On Saturday, 4 February 2017, 1:30, Andrew Gaul <ga...@apache.org> wrote:
>
>
> We rewrote multi-part uploads in jclouds 2.0. You should pass an
> ExecutorService and via PutOptions in your call to BlobStore.putBlob.
>
> On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote:
> > Hi
> >
> > Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart uploads are parallel now ?
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 18:39, Archana C <ca...@yahoo.co.uk> wrote:
> >
> >
> > Thanks it helped
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 12:06, Ignasi Barrera <na...@apache.org> wrote:
> >
> >
> > It looks like the OOM exception is thrown when writing the wire logs. When using the blob store apis you might see binary data in the logs, as the "jclouds.wire" logger logs the response/request payloads which might be huge for some blobs and can cause this kind of exceptions.
> > Could you try disabling the wire logs? (I recommend doing this for production environments).
> > Perhaps for your use case the "jclouds.headers" are enough; that will log all request/reponse path and headers but skip the bodies.
> > More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
> > https://issues.apache.org/jira/browse/JCLOUDS-932
> >
> >
> > HTH!
> > I.
> > On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk> wrote:
> >
> > Hi
> >
> > I have written a sample code for multipart upload using jClouds-2.0
> > Properties overrides = new Properties();
> > BlobStoreContext context = ContextBuilder.newBuilder(" openstack-swift")
> > .endpoint("http://x.xxx.xx.xx: 5000/v2.0")
> > .credentials("xx:xx", "xx")
> > .overrides(overrides)
> > .modules(modules)
> > .buildView(BlobStoreContext. class);
> > BlobStore blobStore = context.getBlobStore();
> > blobStore. createContainerInLocation( null, CONTAINER_NAME);
> > Path path = Paths.get("test2");
> > File f = new File("test2");
> > byte []byteArray = Files.readAllBytes(path);
> > Payload payload = newByteSourcePayload(wrap( byteArray));
> > PutOptions opt = new PutOptions();
> > opt.multipart();
> > Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
> > .payload(payload). contentLength(f.length())
> > .build();
> > String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
> > test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
> > 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> > 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09 d4
> > 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Receiving response 1344471693: HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
> > 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41- 0058940c68
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
> > 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> > 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> > 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
> > 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
> > 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over limit 3145728/262144: wrote temp file
> > 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util. concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
> > at com.google.common.util. concurrent.Futures. wrapAndThrowUnchecked(Futures. java:1380)
> > at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: 1373)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:650)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> > ... 3 more
> > Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ CombinedFuture setExceptionAndMaybeLog
> > SEVERE: input future failed.
> > java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> >
> > Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
> > Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
> >
> > RegardsArchana
> >
> >
> >
> >
> >
> >
> >
> >
>
> --
> Andrew Gaul
> http://gaul.org/
>
>
>
--
Andrew Gaul
http://gaul.org/
Re: jClouds 2.0 MultiPart Upload
Posted by Archana C <ca...@yahoo.co.uk>.
Hi
I think the question was not clear. Parallel upload of multiple file is fine and that can be achieved by using executorservice.
The question here is, does multipartUpload i.e uploading of each part is happening in parallel ?
Does sequential upload of part deprecated ?
RegardsArchana
On Saturday, 4 February 2017, 1:30, Andrew Gaul <ga...@apache.org> wrote:
We rewrote multi-part uploads in jclouds 2.0. You should pass an
ExecutorService and via PutOptions in your call to BlobStore.putBlob.
On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote:
> Hi
>
> Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart uploads are parallel now ?
> RegardsArchana
>
> On Friday, 3 February 2017, 18:39, Archana C <ca...@yahoo.co.uk> wrote:
>
>
> Thanks it helped
> RegardsArchana
>
> On Friday, 3 February 2017, 12:06, Ignasi Barrera <na...@apache.org> wrote:
>
>
> It looks like the OOM exception is thrown when writing the wire logs. When using the blob store apis you might see binary data in the logs, as the "jclouds.wire" logger logs the response/request payloads which might be huge for some blobs and can cause this kind of exceptions.
> Could you try disabling the wire logs? (I recommend doing this for production environments).
> Perhaps for your use case the "jclouds.headers" are enough; that will log all request/reponse path and headers but skip the bodies.
> More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
> https://issues.apache.org/jira/browse/JCLOUDS-932
>
>
> HTH!
> I.
> On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk> wrote:
>
> Hi
>
> I have written a sample code for multipart upload using jClouds-2.0
> Properties overrides = new Properties();
> BlobStoreContext context = ContextBuilder.newBuilder(" openstack-swift")
> .endpoint("http://x.xxx.xx.xx: 5000/v2.0")
> .credentials("xx:xx", "xx")
> .overrides(overrides)
> .modules(modules)
> .buildView(BlobStoreContext. class);
> BlobStore blobStore = context.getBlobStore();
> blobStore. createContainerInLocation( null, CONTAINER_NAME);
> Path path = Paths.get("test2");
> File f = new File("test2");
> byte []byteArray = Files.readAllBytes(path);
> Payload payload = newByteSourcePayload(wrap( byteArray));
> PutOptions opt = new PutOptions();
> opt.multipart();
> Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
> .payload(payload). contentLength(f.length())
> .build();
> String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
> test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
> 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09 d4
> 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Receiving response 1344471693: HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
> 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41- 0058940c68
> 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
> 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
> 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
> 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over limit 3145728/262144: wrote temp file
> 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util. concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
> at com.google.common.util. concurrent.Futures. wrapAndThrowUnchecked(Futures. java:1380)
> at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: 1373)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:650)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> at jclouds20.App.main(App.java: 83)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> at java.lang.StringBuilder. append(StringBuilder.java:208)
> at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> at com.sun.proxy.$Proxy68.put( Unknown Source)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> at java.lang.Thread.run(Thread. java:798)
> at java.lang.Thread. getStackTraceImpl(Native Method)
> at java.lang.Thread. getStackTrace(Thread.java: 1133)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> ... 3 more
> Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ CombinedFuture setExceptionAndMaybeLog
> SEVERE: input future failed.
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> at java.lang.StringBuilder. append(StringBuilder.java:208)
> at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> at com.sun.proxy.$Proxy68.put( Unknown Source)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> at java.lang.Thread.run(Thread. java:798)
> at java.lang.Thread. getStackTraceImpl(Native Method)
> at java.lang.Thread. getStackTrace(Thread.java: 1133)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> at jclouds20.App.main(App.java: 83)
>
> Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
> Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
>
> RegardsArchana
>
>
>
>
>
>
>
>
--
Andrew Gaul
http://gaul.org/
Re: jClouds 2.0 MultiPart Upload
Posted by Andrew Gaul <ga...@apache.org>.
We rewrote multi-part uploads in jclouds 2.0. You should pass an
ExecutorService and via PutOptions in your call to BlobStore.putBlob.
On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote:
> Hi
>
> Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart uploads are parallel now ?
> RegardsArchana
>
> On Friday, 3 February 2017, 18:39, Archana C <ca...@yahoo.co.uk> wrote:
>
>
> Thanks it helped
> RegardsArchana
>
> On Friday, 3 February 2017, 12:06, Ignasi Barrera <na...@apache.org> wrote:
>
>
> It looks like the OOM exception is thrown when writing the wire logs. When using the blob store apis you might see binary data in the logs, as the "jclouds.wire" logger logs the response/request payloads which might be huge for some blobs and can cause this kind of exceptions.
> Could you try disabling the wire logs? (I recommend doing this for production environments).
> Perhaps for your use case the "jclouds.headers" are enough; that will log all request/reponse path and headers but skip the bodies.
> More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
> https://issues.apache.org/jira/browse/JCLOUDS-932
>
>
> HTH!
> I.
> On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk> wrote:
>
> Hi
>
> I have written a sample code for multipart upload using jClouds-2.0
> Properties overrides = new Properties();
> BlobStoreContext context = ContextBuilder.newBuilder(" openstack-swift")
> .endpoint("http://x.xxx.xx.xx: 5000/v2.0")
> .credentials("xx:xx", "xx")
> .overrides(overrides)
> .modules(modules)
> .buildView(BlobStoreContext. class);
> BlobStore blobStore = context.getBlobStore();
> blobStore. createContainerInLocation( null, CONTAINER_NAME);
> Path path = Paths.get("test2");
> File f = new File("test2");
> byte []byteArray = Files.readAllBytes(path);
> Payload payload = newByteSourcePayload(wrap( byteArray));
> PutOptions opt = new PutOptions();
> opt.multipart();
> Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
> .payload(payload). contentLength(f.length())
> .build();
> String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
> test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
> 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09 d4
> 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Receiving response 1344471693: HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
> 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41- 0058940c68
> 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
> 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
> 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
> 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
> 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over limit 3145728/262144: wrote temp file
> 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util. concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
> at com.google.common.util. concurrent.Futures. wrapAndThrowUnchecked(Futures. java:1380)
> at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: 1373)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:650)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> at jclouds20.App.main(App.java: 83)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> at java.lang.StringBuilder. append(StringBuilder.java:208)
> at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> at com.sun.proxy.$Proxy68.put( Unknown Source)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> at java.lang.Thread.run(Thread. java:798)
> at java.lang.Thread. getStackTraceImpl(Native Method)
> at java.lang.Thread. getStackTrace(Thread.java: 1133)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> ... 3 more
> Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ CombinedFuture setExceptionAndMaybeLog
> SEVERE: input future failed.
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> at java.lang.StringBuilder. append(StringBuilder.java:208)
> at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
> at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
> at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
> at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
> at com.sun.proxy.$Proxy68.put( Unknown Source)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
> at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
> at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
> at java.lang.Thread.run(Thread. java:798)
> at java.lang.Thread. getStackTraceImpl(Native Method)
> at java.lang.Thread. getStackTrace(Thread.java: 1133)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
> at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> at jclouds20.App.main(App.java: 83)
>
> Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
> Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
>
> RegardsArchana
>
>
>
>
>
>
>
>
--
Andrew Gaul
http://gaul.org/
Re: jClouds 2.0 MultiPart Upload
Posted by Archana C <ca...@yahoo.co.uk>.
Hi
Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart uploads are parallel now ?
RegardsArchana
On Friday, 3 February 2017, 18:39, Archana C <ca...@yahoo.co.uk> wrote:
Thanks it helped
RegardsArchana
On Friday, 3 February 2017, 12:06, Ignasi Barrera <na...@apache.org> wrote:
It looks like the OOM exception is thrown when writing the wire logs. When using the blob store apis you might see binary data in the logs, as the "jclouds.wire" logger logs the response/request payloads which might be huge for some blobs and can cause this kind of exceptions.
Could you try disabling the wire logs? (I recommend doing this for production environments).
Perhaps for your use case the "jclouds.headers" are enough; that will log all request/reponse path and headers but skip the bodies.
More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
https://issues.apache.org/jira/browse/JCLOUDS-932
HTH!
I.
On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk> wrote:
Hi
I have written a sample code for multipart upload using jClouds-2.0
Properties overrides = new Properties();
BlobStoreContext context = ContextBuilder.newBuilder(" openstack-swift")
.endpoint("http://x.xxx.xx.xx: 5000/v2.0")
.credentials("xx:xx", "xx")
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext. class);
BlobStore blobStore = context.getBlobStore();
blobStore. createContainerInLocation( null, CONTAINER_NAME);
Path path = Paths.get("test2");
File f = new File("test2");
byte []byteArray = Files.readAllBytes(path);
Payload payload = newByteSourcePayload(wrap( byteArray));
PutOptions opt = new PutOptions();
opt.multipart();
Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
.payload(payload). contentLength(f.length())
.build();
String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09 d4
10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Receiving response 1344471693: HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41- 0058940c68
10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over limit 3145728/262144: wrote temp file
10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util. concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at com.google.common.util. concurrent.Futures. wrapAndThrowUnchecked(Futures. java:1380)
at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: 1373)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:650)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
at jclouds20.App.main(App.java: 83)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
at java.lang.StringBuilder. append(StringBuilder.java:208)
at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
at org.jclouds.logging.internal. Wire.output(Wire.java:176)
at org.jclouds.logging.internal. Wire.output(Wire.java:143)
at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
at com.sun.proxy.$Proxy68.put( Unknown Source)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
at java.util.concurrent. FutureTask.run(FutureTask. java:274)
at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread. java:798)
at java.lang.Thread. getStackTraceImpl(Native Method)
at java.lang.Thread. getStackTrace(Thread.java: 1133)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
... 3 more
Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ CombinedFuture setExceptionAndMaybeLog
SEVERE: input future failed.
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
at java.lang.StringBuilder. append(StringBuilder.java:208)
at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
at org.jclouds.logging.internal. Wire.output(Wire.java:176)
at org.jclouds.logging.internal. Wire.output(Wire.java:143)
at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
at com.sun.proxy.$Proxy68.put( Unknown Source)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
at java.util.concurrent. FutureTask.run(FutureTask. java:274)
at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread. java:798)
at java.lang.Thread. getStackTraceImpl(Native Method)
at java.lang.Thread. getStackTrace(Thread.java: 1133)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
at jclouds20.App.main(App.java: 83)
Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
RegardsArchana
Re: jClouds 2.0 MultiPart Upload
Posted by Archana C <ca...@yahoo.co.uk>.
Thanks it helped
RegardsArchana
On Friday, 3 February 2017, 12:06, Ignasi Barrera <na...@apache.org> wrote:
It looks like the OOM exception is thrown when writing the wire logs. When using the blob store apis you might see binary data in the logs, as the "jclouds.wire" logger logs the response/request payloads which might be huge for some blobs and can cause this kind of exceptions.
Could you try disabling the wire logs? (I recommend doing this for production environments).
Perhaps for your use case the "jclouds.headers" are enough; that will log all request/reponse path and headers but skip the bodies.
More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
https://issues.apache.org/jira/browse/JCLOUDS-932
HTH!
I.
On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk> wrote:
Hi
I have written a sample code for multipart upload using jClouds-2.0
Properties overrides = new Properties();
BlobStoreContext context = ContextBuilder.newBuilder(" openstack-swift")
.endpoint("http://x.xxx.xx.xx: 5000/v2.0")
.credentials("xx:xx", "xx")
.overrides(overrides)
.modules(modules)
.buildView(BlobStoreContext. class);
BlobStore blobStore = context.getBlobStore();
blobStore. createContainerInLocation( null, CONTAINER_NAME);
Path path = Paths.get("test2");
File f = new File("test2");
byte []byteArray = Files.readAllBytes(path);
Payload payload = newByteSourcePayload(wrap( byteArray));
PutOptions opt = new PutOptions();
opt.multipart();
Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
.payload(payload). contentLength(f.length())
.build();
String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
test2 is the file I am trying to upload which is of size 36MB and I am getting the following exception
10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: fd72b74db90c46cabcca3f317d5a09 d4
10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Receiving response 1344471693: HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 GMT
10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: tx83ba6249347c43c99bb41- 0058940c68
10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; charset=UTF-8
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 -------> Container Creation Successful
10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> invoking object:put
10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over limit 3145728/262144: wrote temp file
10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over limit 33554432/262144: wrote temp fileException in thread "main" com.google.common.util. concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at com.google.common.util. concurrent.Futures. wrapAndThrowUnchecked(Futures. java:1380)
at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: 1373)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:650)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
at jclouds20.App.main(App.java: 83)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
at java.lang.StringBuilder. append(StringBuilder.java:208)
at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
at org.jclouds.logging.internal. Wire.output(Wire.java:176)
at org.jclouds.logging.internal. Wire.output(Wire.java:143)
at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
at com.sun.proxy.$Proxy68.put( Unknown Source)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
at java.util.concurrent. FutureTask.run(FutureTask. java:274)
at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread. java:798)
at java.lang.Thread. getStackTraceImpl(Native Method)
at java.lang.Thread. getStackTrace(Thread.java: 1133)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
... 3 more
Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ CombinedFuture setExceptionAndMaybeLog
SEVERE: input future failed.
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
at java.lang.StringBuilder. append(StringBuilder.java:208)
at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
at org.jclouds.logging.internal. Wire.output(Wire.java:176)
at org.jclouds.logging.internal. Wire.output(Wire.java:143)
at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( BaseHttpCommandExecutorService .java:97)
at org.jclouds.rest.internal. InvokeHttpMethod.invoke( InvokeHttpMethod.java:90)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:73)
at org.jclouds.rest.internal. InvokeHttpMethod.apply( InvokeHttpMethod.java:44)
at org.jclouds.reflect. FunctionalReflection$ FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117)
at com.google.common.reflect. AbstractInvocationHandler. invoke( AbstractInvocationHandler. java:87)
at com.sun.proxy.$Proxy68.put( Unknown Source)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. uploadMultipartPart( RegionScopedSwiftBlobStore. java:529)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:666)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. java:653)
at java.util.concurrent. FutureTask.run(FutureTask. java:274)
at java.util.concurrent. ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1157)
at java.util.concurrent. ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread. java:798)
at java.lang.Thread. getStackTraceImpl(Native Method)
at java.lang.Thread. getStackTrace(Thread.java: 1133)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:647)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. java:628)
at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
at jclouds20.App.main(App.java: 83)
Should putBlob with multipart enabled in PutOptions not be used for uploading files greater than 32MB from jClouds-2.0 ?
Is it mandatory to use Swift API's initiate, upload and complete multipart from jClouds-2.0 ?
RegardsArchana
Re: jClouds 2.0 MultiPart Upload
Posted by Ignasi Barrera <na...@apache.org>.
It looks like the OOM exception is thrown when writing the wire logs. When
using the blob store apis you might see binary data in the logs, as the
"jclouds.wire" logger logs the response/request payloads which might be
huge for some blobs and can cause this kind of exceptions.
Could you try disabling the wire logs? (I recommend doing this for
production environments).
Perhaps for your use case the "jclouds.headers" are enough; that will log
all request/reponse path and headers but skip the bodies.
More on this here:
https://issues.apache.org/jira/browse/JCLOUDS-1187
https://issues.apache.org/jira/browse/JCLOUDS-932
HTH!
I.
On Feb 3, 2017 06:22, "Archana C" <ca...@yahoo.co.uk> wrote:
> Hi
>
> *I have written a sample code for multipart upload using jClouds-2.0*
>
> Properties overrides = new Properties();
> BlobStoreContext context = ContextBuilder.newBuilder("
> openstack-swift")
> .endpoint("http://x.xxx.xx.xx:5000/v2.0")
> .credentials("xx:xx", "xx")
> .overrides(overrides)
> .modules(modules)
> .buildView(BlobStoreContext.class);
> BlobStore blobStore = context.getBlobStore();
> blobStore.createContainerInLocation(null, CONTAINER_NAME);
> Path path = Paths.get("test2");
> File f = new File("test2");
> byte []byteArray = Files.readAllBytes(path);
> Payload payload = newByteSourcePayload(wrap(byteArray));
> PutOptions opt = new PutOptions();
> opt.multipart();
> Blob blob = blobStore.blobBuilder(OBJECT_NAME)
> .payload(payload).contentLength(f.length())
> .build();
> String etag = blobStore.putBlob(CONTAINER_NAME, blob, opt);
>
> *test2 is the file I am trying to upload which is of size 36MB and I am
> getting the following exception*
>
> 10:21:52.355 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService -
> Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_
> 0909ac10e7024847b1a9fe9787c7de8f/arctestMP HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT
> http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP
> HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token:
> fd72b74db90c46cabcca3f317d5a09d4
> 10:21:53.129 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService -
> Receiving response 1344471693: HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017
> 04:51:53 GMT
> 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id:
> tx83ba6249347c43c99bb41-0058940c68
> 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html;
> charset=UTF-8
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0
> -------> *Container Creation Successful*
> 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal.InvokeHttpMethod -
> >> invoking object:put
> 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal.InvokeHttpMethod -
> >> invoking object:put
> 10:21:53.374 [user thread 1] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
> - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_
> 0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/
> 1486097513.327000/0/33554432/00000001 HTTP/1.1
> 10:21:53.376 [user thread 0] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
> - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_
> 0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/
> 1486097513.327000/0/33554432/00000000 HTTP/1.1
> 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal.HttpWire -
> over limit* 3145728/262144*: wrote temp file
> 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal.HttpWire -
> over limit* 33554432/262144*: wrote temp file
> Exception in thread "main" com.google.common.util.concurrent.ExecutionError:
> java.lang.OutOfMemoryError: Java heap space
> at com.google.common.util.concurrent.Futures.
> wrapAndThrowUnchecked(Futures.java:1380)
> at com.google.common.util.concurrent.Futures.
> getUnchecked(Futures.java:1373)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.
> java:650)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.
> java:628)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putBlob(RegionScopedSwiftBlobStore.java:274)
> at jclouds20.App.main(App.java:83)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringBuilder.ensureCapacityImpl(StringBuilder.java:342)
> at java.lang.StringBuilder.append(StringBuilder.java:208)
> at org.jclouds.logging.internal.Wire.wire(Wire.java:68)
> at org.jclouds.logging.internal.Wire.copy(Wire.java:99)
> at org.jclouds.logging.internal.Wire.output(Wire.java:176)
> at org.jclouds.logging.internal.Wire.output(Wire.java:143)
> at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:296)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(
> BaseHttpCommandExecutorService.java:97)
> at org.jclouds.rest.internal.InvokeHttpMethod.invoke(
> InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(
> InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(
> InvokeHttpMethod.java:44)
> at org.jclouds.reflect.FunctionalReflection$
> FunctionalInvocationHandler.handleInvocation(
> FunctionalReflection.java:117)
> at com.google.common.reflect.AbstractInvocationHandler.invoke(
> AbstractInvocationHandler.java:87)
> at com.sun.proxy.$Proxy68.put(Unknown Source)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.uploadMultipartPart(RegionScopedSwiftBlobStore.
> java:529)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.
> java:666)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.
> java:653)
> at java.util.concurrent.FutureTask.run(FutureTask.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1157)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:627)
> at java.lang.Thread.run(Thread.java:798)
> at java.lang.Thread.getStackTraceImpl(Native Method)
> at java.lang.Thread.getStackTrace(Thread.java:1133)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.
> java:647)
> ... 3 more
> Feb 03, 2017 10:22:03 AM com.google.common.util.concurrent.Futures$CombinedFuture
> setExceptionAndMaybeLog
> SEVERE: input future failed.
> java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringBuilder.ensureCapacityImpl(StringBuilder.java:342)
> at java.lang.StringBuilder.append(StringBuilder.java:208)
> at org.jclouds.logging.internal.Wire.wire(Wire.java:68)
> at org.jclouds.logging.internal.Wire.copy(Wire.java:99)
> at org.jclouds.logging.internal.Wire.output(Wire.java:176)
> at org.jclouds.logging.internal.Wire.output(Wire.java:143)
> at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:296)
> at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(
> BaseHttpCommandExecutorService.java:97)
> at org.jclouds.rest.internal.InvokeHttpMethod.invoke(
> InvokeHttpMethod.java:90)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(
> InvokeHttpMethod.java:73)
> at org.jclouds.rest.internal.InvokeHttpMethod.apply(
> InvokeHttpMethod.java:44)
> at org.jclouds.reflect.FunctionalReflection$
> FunctionalInvocationHandler.handleInvocation(
> FunctionalReflection.java:117)
> at com.google.common.reflect.AbstractInvocationHandler.invoke(
> AbstractInvocationHandler.java:87)
> at com.sun.proxy.$Proxy68.put(Unknown Source)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.uploadMultipartPart(RegionScopedSwiftBlobStore.
> java:529)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.
> java:666)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore$BlobUploader.call(RegionScopedSwiftBlobStore.
> java:653)
> at java.util.concurrent.FutureTask.run(FutureTask.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1157)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:627)
> at java.lang.Thread.run(Thread.java:798)
> at java.lang.Thread.getStackTraceImpl(Native Method)
> at java.lang.Thread.getStackTrace(Thread.java:1133)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.
> java:647)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.
> java:628)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putBlob(RegionScopedSwiftBlobStore.java:274)
> at jclouds20.App.main(App.java:83)
>
>
> *Should putBlob with multipart enabled in PutOptions not be used for
> uploading files greater than 32MB from jClouds-2.0 ?*
>
> *Is it mandatory to use Swift API's initiate, upload and complete
> multipart from jClouds-2.0 ?*
>
> Regards
> Archana
>
>
>