You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mingliang Liu (Jira)" <ji...@apache.org> on 2021/01/25 18:55:00 UTC
[jira] [Resolved] (HADOOP-16355) ZookeeperMetadataStore: Use
Zookeeper as S3Guard backend store
[ https://issues.apache.org/jira/browse/HADOOP-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingliang Liu resolved HADOOP-16355.
------------------------------------
Resolution: Abandoned
As documented by [[HADOOP-17480]], AWS S3 is consistent and that S3Guard is not needed.
> ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store
> --------------------------------------------------------------
>
> Key: HADOOP-16355
> URL: https://issues.apache.org/jira/browse/HADOOP-16355
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Mingliang Liu
> Priority: Major
>
> When S3Guard was proposed, there are a couple of valid reasons to choose DynamoDB as its default backend store: 0) seamless integration as part of AWS ecosystem e.g. client library 1) it's a managed web service which is zero operational cost, highly available and infinitely scalable 2) it's performant with single digit latency 3) it's proven by Netflix's S3mper (not actively maintained) and EMRFS (closed source and usage). As it's pluggable, it's possible to implement {{MetadataStore}} with other backend store without changing semantics, besides null and in-memory local ones.
> Here we propose {{ZookeeperMetadataStore}} which uses Zookeeper as S3Guard backend store. Its main motivation is to provide a new MetadataStore option which:
> # can be easily integrated as Zookeeper is heavily used in Hadoop community
> # affordable performance as both client and Zookeeper ensemble are usually "local" in a Hadoop cluster (ZK/HBase/Hive etc)
> # removes DynamoDB dependency
> Obviously all use cases will not prefer this to default DynamoDB store. For e.g. ZK might not scale well if there are dozens of S3 buckets and each has millions of objects. Our use case is targeting HBase to store HFiles on S3 instead of HDFS. A total solution for HBase on S3 must be HBOSS (see HBASE-22149) for recovering atomicity of metadata operations like rename, and S3Guard for consistent enumeration and access to object store bucket metadata. We would like to use Zookeeper as backend store for both.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org