putBlob failing, but acting like success

[jo...@gmail.com, jo...@gmail.com]

I'm calling putBlob to put a blob to an S3 ninja server. The server is returning a ridiculous 500 server error response page - it contains a ton of javascript and the jclouds SAX parser is choking on the content of a <script> tag (not surprising). The parser exception is properly caught in AWSUtils.parseAWSErrorFromContent, where it logs the issue and returns null for the AWSError. Tracing it up the stack a bit, we get to BaseHttpCommandExecutorService.shouldContinue, where it properly interprets false from the retryHandler.shouldRetryRequest method as "nope" and (again properly) calls errorHandler.handleError(command, response). 

However, errorHandler.handleError isn't actually doing anything visible from my perspective as the guy who calls putBlob. The return value of putBlob is supposed to be the etag for services that provide an etag in the response or null otherwise. So there's no way I can tell I got an error from looking at the return value (is there)?

I would have assumed that errorHandler (which ends up being serverErrorHandler via DelegatingErrorHandler) .handleError would end up throwing an exception that I could catch in my code.

FYI - here's my client code:

                try (InputStream is = new ByteBufferBackedInputStream(content)) {
                    BlobStore blobStore = context.getBlobStore();
                    BlobBuilder.PayloadBlobBuilder blobBuilder =
                            blobStore.blobBuilder(key).payload(is);

                    // set content-md5 to allow the server to validate the uploaded data
                    byte[] checksum = meta.getChecksum();
                    if (checksum != null) {
                        blobBuilder.contentMD5(HashCode.fromBytes(checksum));
                    }

                    // set content-length so the server knows when to stop reading
                    long contentLength = content.remaining();
                    Blob blob = blobBuilder.contentLength(contentLength)
                            .userMetadata(meta.getUserMeta()).build();

                    // upload it - ignore etag - we provided checksum in metadata - server will validate
                    blobStore.putBlob(cspInfo.getContainer(), blob);
                    return true;
                } catch (ContainerNotFoundException ex) {
                    log.error("upload cloud object {} failed; container not found: ", key, ex);
                    return false;
                } catch (Exception ex) {
                    log.error("upload cloud object {} failed due to {}; retrying {} more times: {}",
                            key, ex, retries, Throwables.getStackTraceAsString(ex));
                    content.reset();
                }

As you can see, I catch everything, and I'm not catching anything here. I'm using jclouds 2.1.0, btw.

Thanks,
John

Re: putBlob failing, but acting like success

[jo...@gmail.com, jo...@gmail.com]

Nevermind - egg on face. There was an exception caught in my code, but it was buried in the middle of the HttpResponseException content so I missed it. The problem turned out to be one where s3 ninja was literally returning success without having written the file due to a previous error where it had run out of file handles and committed a key without having actually written it to disk.

I added code to my client to check the returned etag against my own checksum. I check to be sure both remote and local md5 sums are available before doing this comparison, but most services return an etag, so I'm not worried.

Thanks for your patience Andrew. 

Regards,
John

Re: putBlob failing, but acting like success

[Andrew Gaul <ga...@apache.org>]

On Fri, Aug 17, 2018 at 10:04:29PM -0000, john.calcote@gmail.com wrote:
> I'm calling putBlob to put a blob to an S3 ninja server. The server is returning a ridiculous 500 server error response page - it contains a ton of javascript and the jclouds SAX parser is choking on the content of a <script> tag (not surprising). The parser exception is properly caught in AWSUtils.parseAWSErrorFromContent, where it logs the issue and returns null for the AWSError. Tracing it up the stack a bit, we get to BaseHttpCommandExecutorService.shouldContinue, where it properly interprets false from the retryHandler.shouldRetryRequest method as "nope" and (again properly) calls errorHandler.handleError(command, response). 
> 
> However, errorHandler.handleError isn't actually doing anything visible from my perspective as the guy who calls putBlob. The return value of putBlob is supposed to be the etag for services that provide an etag in the response or null otherwise. So there's no way I can tell I got an error from looking at the return value (is there)?
> 
> I would have assumed that errorHandler (which ends up being serverErrorHandler via DelegatingErrorHandler) .handleError would end up throwing an exception that I could catch in my code.

I agree with your assumptions; I expect to see putBlob throw
HttpResponseException in this situation.  Returning null from
AWSUtils.parseAWSErrorFromContent seems reasonable albeit not
documented.  Could you add some logging to
ParseAWSErrorFromXmlContent.handleError to see where it swallows the
error code?  Also could you open an issue in Jira so we can track this
defect?

-- 
Andrew Gaul
http://gaul.org/