[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

["Aaron Fabbri (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847856#comment-15847856 ] 

Aaron Fabbri commented on HADOOP-13904:
---------------------------------------

Thank you for the helpful review [~stevel@apache.org]

{quote}
do these run iff `-Dscale` is set? 
{quote}

Yep.. They extend {{S3AScaleTestBase}}.

{quote}
clearMetadataStore(ms, count); may need to go into a finally clause of testMoves()
{quote}

Good call.  I'll do the same in testPut() as well.  I'll retest with those changes and get you timing details as well.

These could run in parallel if we ensured they had their own DDB table to work with.  I think it would be more simple just to mark them as serial for scale tests.  I'll add that as well.








> DynamoDBMetadataStore to handle DDB throttling failures through retry policy
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-13904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13904
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Steve Loughran
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13904-HADOOP-13345.001.patch
>
>
> When you overload DDB, you get error messages warning of throttling, [as documented by AWS|http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.MessagesAndCodes]
> Reduce load on DDB by doing a table lookup before the create, then, in table create/delete operations and in get/put actions, recognise the error codes and retry using an appropriate retry policy (exponential backoff + ultimate failure) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org