You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by "Xavier BOURGOUIN (Jira)" <ji...@apache.org> on 2019/10/10 15:48:00 UTC
[jira] [Commented] (JCLOUDS-1505) BlobStore.blobMetadata(container,
object) returns a StorageMetadata object with empty size when using
org.jclouds.http.apachehc.config.ApacheHCHttpCommandExecutorServiceModule
[ https://issues.apache.org/jira/browse/JCLOUDS-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948716#comment-16948716 ]
Xavier BOURGOUIN commented on JCLOUDS-1505:
-------------------------------------------
I've made a proposal fix for this issue: https://github.com/apache/jclouds/pull/48
Could you please review it when time permits ?
Thanks!
Brgds,
Xavier
> BlobStore.blobMetadata(container, object) returns a StorageMetadata object with empty size when using org.jclouds.http.apachehc.config.ApacheHCHttpCommandExecutorServiceModule
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: JCLOUDS-1505
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1505
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-drivers
> Reporter: Xavier BOURGOUIN
> Priority: Major
> Attachments: jclouds_null_size_reproducer.tar.gz
>
>
> Hello,
>
> When calling *blobStore.blobMetadata(container, objectName).getSize()* with a blobStoreContext using ApacheHCHttpCommandExecutorServiceModule, the returned BlobMetadata object has a size attribute which is null.
> Issue spotted while using jclouds 2.1.2 over an S3 object store (Scality, but seems like it is reproducible with any other S3 provider, and potentially others OS providers like GCS, to be confirmed).
> In essence, the minimal reproducer pseudo-code is:
> {code:java}
> BlobStoreContext context = ContextBuilder.newBuilder(provider)
> .credentials(identity, credential)
> .endpoint(endpoint)
> // usage of ApacheHCHttp module
> .modules(ImmutableList.<Module> of(new ApacheHCHttpCommandExecutorServiceModule(), new SLF4JLoggingModule()))
> .buildView(BlobStoreContext.class);
>
> try {
> BlobStore blobStore = context.getBlobStore();
> // output is: "File size from metadata: null"
> System.out.println("File size from metadata: " + blobStore.blobMetadata(DUMMY_CONTAINER, DUMMY_FILE_NAME).getSize());
> } finally {
> context.close();
> }
> {code}
>
> For convenience, I've attached a ready to run reproducer to this ticket, here's how to run it:
>
> {code:bash}
> > mkdir reproducer
> > tar xvzf jclouds_null_size_reproducer.tar.gz -C reproducer
> > cd reproducer
> > mvn install
> # note: pre-req from this stage is to have an S3 Object Store available, you can use minio one if you have docker (docker run -it -p 9000:9000 -v /mnt/data:/data minio/minio server /data)
> # replace with your Object Store endpoint, identity and access key
> > java -jar target/blobstore-basics-jar-with-dependencies.jar s3 http://10.66.33.1:9000 <identity here> <access key here>
> File size from metadata: 8
> File size from metadata (with ApacheHCHttpCommandExecutorServiceModule): null
> {code}
> The reproducer code has jclouds-wire logs enabled (inside log folder) and I've also added the log files inside the attached tarball for convenience. We see that in the HTTP HEAD response (the 2nd one, which correspond to my .blobMetadata() call using ApacheHCHttpCommandExecutorServiceModule), the Content-* headers are missing :
> {code:bash}
> >cat log/jclouds-wire.log
> # note: first .blobMetadata() call without ApacheHCHttpCommandExecutorServiceModule, the response contains Content-* headers
> 2019-07-31 17:30:13,257 DEBUG [jclouds.headers] [main] >> HEAD http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1
> 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul 2019 15:30:13 GMT
> 2019-07-31 17:30:13,258 DEBUG [jclouds.headers] [main] >> Authorization: AWS 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE=
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Server: MinIO/RELEASE.2019-07-24T02-02-23Z
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << ETag: "35a41457219c775659b6018fb7f465a1-1"
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Xss-Protection: 1; mode=block
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Security-Policy: block-all-mixed-content
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Vary: Origin
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, 31 Jul 2019 15:30:13 GMT
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul 2019 15:30:13 GMT
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: 15B687995977CC49
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Type: application/unknown
> 2019-07-31 17:30:13,259 DEBUG [jclouds.headers] [main] << Content-Length: 8
> # note: second call with ApacheHCHttpCommandExecutorServiceModule, the reponse is missing the content-* headers
> 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> HEAD http://10.66.33.1:9000/foo/example_for_empty_size HTTP/1.1
> 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Date: Wed, 31 Jul 2019 15:30:13 GMT
> 2019-07-31 17:30:13,355 DEBUG [jclouds.headers] [main] >> Authorization: AWS 8A0CESVSLMU5HSSBIOB0:hvzLzlfV+ep7K6Om+tfFss5AuZE=
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << HTTP/1.1 200 OK
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Accept-Ranges: bytes
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Content-Security-Policy: block-all-mixed-content
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << ETag: "35a41457219c775659b6018fb7f465a1-1"
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Last-Modified: Wed, 31 Jul 2019 15:30:13 GMT
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Server: MinIO/RELEASE.2019-07-24T02-02-23Z
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Vary: Origin
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Amz-Request-Id: 15B6879961644613
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << X-Xss-Protection: 1; mode=block
> 2019-07-31 17:30:13,396 DEBUG [jclouds.headers] [main] << Date: Wed, 31 Jul 2019 15:30:13 GMT
> {code}
>
> I suspect the issue is coming from [ApacheHCHttpCommandExecutorService.java#invoke|https://github.com/apache/jclouds/blob/master/drivers/apachehc/src/main/java/org/jclouds/http/apachehc/ApacheHCHttpCommandExecutorService.java#L87] method, it seems to be expecting the content-* headers are exclusively coming from the reponse.getEntity() (which I believe is null in case of a response to a HEAD request), because it strips it out of the headers list (which are well containing the content-* headers before the call to "filterOutContentHeaders"). A possible fix to cope with all requests kinds could be to strip it out *only* if it is already part of the payload (which I guess was the initial intent).
>
> Environment details:
> * jclouds version: 2.1.2 (but it is probably reproducible on older versions, to be confirmed)
> * Operating system: Linux (kernel 4.15.0-55)
> * Storage provider: S3 (reproduced on both Scality and MinIO), probably reproducible with other storage providers involving HTTP HEAD requests to get the metadata (GCS, ..)
> * Java version:
> {code:java}
> >java -version{code}
> *
> {code:java}
> java version "1.8.0_201"
> Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
> Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
> {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)