[jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Encho Belezirev <no...@github.com>]

This is needed because of this issue: https://issues.apache.org/jira/browse/JCLOUDS-1504
The null check will ignore the BlobMetadata which is not found in the blob store and will not be added to the result from the query.

JIRA issue: https://issues.apache.org/jira/browse/JCLOUDS-1504
You can view, comment on, or merge this pull request online at:

  https://github.com/jclouds/jclouds/pull/1276

-- Commit Summary --

  * Ignore null StorageMetadata returned from blob store

-- File Changes --

    M core/src/main/java/org/jclouds/concurrent/FutureIterables.java (4)

-- Patch Links --

https://github.com/jclouds/jclouds/pull/1276.patch
https://github.com/jclouds/jclouds/pull/1276.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276

Re: [jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Andrew Gaul <no...@github.com>]

Sorry this approach will never merge.  As stated in my previous comment, we need to do a root cause analysis to determine where null S3 blobs come from instead of changing the behavior for all possible callers across all possible providers.  I will share more in [JCLOUDS-1504](https://issues.apache.org/jira/browse/JCLOUDS-1504).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276#issuecomment-520980970

Re: [jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Encho Belezirev <no...@github.com>]

I agree it is not the best solution ever but for your worries, if the metadata for some blobs could not be taken for some reason(failure, exception, etc.) should there be an exception risen and the processing to stop? For me it is not quite ok to return null if something goes wrong :)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276#issuecomment-513652827

Re: [jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Encho Belezirev <no...@github.com>]

Moreover, I have added logs in the JIRA issue. In the logs, you could check that when some blob metadata is not found(the response code is 404 Not Found), it is being included in the list of blobs(which now includes null objects) returned to the user. For me that is not OK, so that I decided to put a null check just for the retrieval of the blobMetadata which is returning the nulls. 

If I understand the behaviour of the s3 client correctly, when something is not found in the blob store, it returns null objects. I am saying this, because all the files which are related to the null metadatas, are being scheduled for deletion in the blob store. 


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276#issuecomment-513654175

Re: [jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Encho Belezirev <no...@github.com>]

Something more, could you tell me where I can find the exact implementation of the aws-s3 client which implements the S3Client interface? I want to check whether the issue is in the client.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276#issuecomment-513695311

Re: [jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Andrew Gaul <no...@github.com>]

Closed #1276.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276#event-2554519508

Re: [jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Encho Belezirev <no...@github.com>]

Hello @gaul,

Any updates here? The issue is hurting our code base.
Let's continue the discussion!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276#issuecomment-520712202

Re: [jclouds/jclouds] Ignore null StorageMetadata returned from blob store (#1276)

[Andrew Gaul <no...@github.com>]

gaul requested changes on this pull request.

Ignoring null elements for all FutureIterables to fix a local problem is blobstore is a hack.  We need to do a root cause analyses to determine what generates these null elements.  For example this may mask some blobs which failed to get metadata.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1276#pullrequestreview-263799704