You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Sameer Choudhary (JIRA)" <ji...@apache.org> on 2017/02/09 00:23:42 UTC
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for
S3A
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858755#comment-15858755 ]
Sameer Choudhary commented on HADOOP-13345:
-------------------------------------------
Hi,
Today, I attended the talk on the project at Spark Summit 2017. Thanks for putting in all the effort!
I have a question regarding pricing of DynamoDB. It charges on read/write request rate. So, users might have to pay high amount of price for getting the consistency guarantees. This would especially affect large Spark Jobs with many parallel executing tasks that are trying to read/write to DynamoDB. Putting throttling will affect the job performance. Some benchmarks here would be great.
A solution could be for S3Guard to additionally support for custom Key Value store such as Apache HBase that supports strictly consistent reads/writes. A user can create a separate cluster or use the same Spark cluster to setup the store. The benefit of the approach is that users can now achieve high throughput on even large Spark jobs with paying just a fraction of cost.
> S3Guard: Improved Consistency for S3A
> -------------------------------------
>
> Key: HADOOP-13345
> URL: https://issues.apache.org/jira/browse/HADOOP-13345
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch, S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf, S3GuardImprovedConsistencyforS3AV2.pdf
>
>
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a stronger consistency model than what is currently offered. The solution coordinates with a strongly consistent external store to resolve inconsistencies caused by the S3 eventual consistency model.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org