You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by ad...@apache.org on 2021/11/18 13:04:02 UTC
[ozone-site] branch asf-site updated: HDDS-6002. Add docs for 1.2.0 (#18)
This is an automated email from the ASF dual-hosted git repository.
adoroszlai pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 869c806 HDDS-6002. Add docs for 1.2.0 (#18)
869c806 is described below
commit 869c80665c770862b58135a70448e615b4ef2b68
Author: Ethan Rose <33...@users.noreply.github.com>
AuthorDate: Thu Nov 18 05:02:10 2021 -0800
HDDS-6002. Add docs for 1.2.0 (#18)
---
docs/1.2.0/NOTES.md | 20 +
docs/1.2.0/OzoneOverview.png | Bin 0 -> 41729 bytes
docs/1.2.0/OzoneOverview.svg | 238 ++++
docs/1.2.0/SCMBlockDiagram.png | Bin 0 -> 14714 bytes
docs/1.2.0/concept.html | 553 +++++++++
docs/1.2.0/concept/ContainerMetadata.png | Bin 0 -> 98493 bytes
docs/1.2.0/concept/Containers.png | Bin 0 -> 24775 bytes
docs/1.2.0/concept/FunctionalOzone.png | Bin 0 -> 55334 bytes
docs/1.2.0/concept/OzoneBlock.png | Bin 0 -> 4650 bytes
docs/1.2.0/concept/OzoneManager-ReadPath.png | Bin 0 -> 81030 bytes
docs/1.2.0/concept/OzoneManager-WritePath.png | Bin 0 -> 96696 bytes
docs/1.2.0/concept/OzoneManager.png | Bin 0 -> 13327 bytes
docs/1.2.0/concept/ReconHighLevelDesign.png | Bin 0 -> 239168 bytes
docs/1.2.0/concept/ReconOmDesign.png | Bin 0 -> 162797 bytes
docs/1.2.0/concept/ReconScmDesign.png | Bin 0 -> 181628 bytes
docs/1.2.0/concept/StorageContainerManager.png | Bin 0 -> 13336 bytes
docs/1.2.0/concept/containers.html | 452 +++++++
docs/1.2.0/concept/datanodes.html | 461 ++++++++
docs/1.2.0/concept/index.xml | 65 ++
docs/1.2.0/concept/overview.html | 461 ++++++++
docs/1.2.0/concept/ozoneBlockDiagram.png | Bin 0 -> 45218 bytes
docs/1.2.0/concept/ozonemanager.html | 578 +++++++++
docs/1.2.0/concept/recon.html | 619 ++++++++++
docs/1.2.0/concept/storagecontainermanager.html | 532 +++++++++
docs/1.2.0/css/bootstrap-theme.min.css | 6 +
docs/1.2.0/css/bootstrap-theme.min.css.map | 1 +
docs/1.2.0/css/bootstrap.min.css | 6 +
docs/1.2.0/css/bootstrap.min.css.map | 1 +
docs/1.2.0/css/ozonedoc.css | 212 ++++
docs/1.2.0/design.html | 649 +++++++++++
docs/1.2.0/design/configless.html | 398 +++++++
docs/1.2.0/design/decommissioning.html | 1230 ++++++++++++++++++++
docs/1.2.0/design/ec.html | 402 +++++++
docs/1.2.0/design/gdpr.html | 396 +++++++
docs/1.2.0/design/index.xml | 245 ++++
docs/1.2.0/design/locks.html | 395 +++++++
docs/1.2.0/design/multiraft.html | 396 +++++++
docs/1.2.0/design/namespace-support.html | 398 +++++++
docs/1.2.0/design/nfs.html | 401 +++++++
docs/1.2.0/design/nonrolling-upgrade.html | 397 +++++++
docs/1.2.0/design/ofs.html | 398 +++++++
docs/1.2.0/design/omha.html | 405 +++++++
docs/1.2.0/design/omprepare.html | 421 +++++++
docs/1.2.0/design/ozone-enhancement-proposals.html | 568 +++++++++
docs/1.2.0/design/recon1.html | 401 +++++++
docs/1.2.0/design/recon2.html | 397 +++++++
docs/1.2.0/design/s3-performance-new.png | Bin 0 -> 33688 bytes
docs/1.2.0/design/s3-performance-old.png | Bin 0 -> 34927 bytes
docs/1.2.0/design/s3-performance.html | 564 +++++++++
docs/1.2.0/design/s3gateway.html | 398 +++++++
docs/1.2.0/design/scmha.html | 404 +++++++
docs/1.2.0/design/storage-class.html | 395 +++++++
docs/1.2.0/design/tde.html | 395 +++++++
docs/1.2.0/design/token.html | 393 +++++++
docs/1.2.0/design/topology.html | 397 +++++++
docs/1.2.0/design/trash.html | 396 +++++++
docs/1.2.0/design/typesafeconfig.html | 397 +++++++
docs/1.2.0/design/upgrade-dev-primer.html | 483 ++++++++
docs/1.2.0/design/volume-management.html | 519 +++++++++
docs/1.2.0/en/index.html | 1 +
docs/1.2.0/en/sitemap.xml | 646 ++++++++++
docs/1.2.0/feature.html | 593 ++++++++++
docs/1.2.0/feature/HA-OM-doublebuffer.png | Bin 0 -> 77661 bytes
docs/1.2.0/feature/HA-OM.png | Bin 0 -> 60888 bytes
docs/1.2.0/feature/PrefixFSO-Delete.png | Bin 0 -> 253792 bytes
docs/1.2.0/feature/PrefixFSO-Format.png | Bin 0 -> 270550 bytes
docs/1.2.0/feature/PrefixFSO-Rename.png | Bin 0 -> 227851 bytes
docs/1.2.0/feature/index.xml | 82 ++
docs/1.2.0/feature/nonrolling-upgrade.html | 505 ++++++++
docs/1.2.0/feature/observability.html | 561 +++++++++
docs/1.2.0/feature/om-ha.html | 475 ++++++++
docs/1.2.0/feature/prefixfso.html | 454 ++++++++
docs/1.2.0/feature/quota.html | 483 ++++++++
docs/1.2.0/feature/recon.html | 431 +++++++
docs/1.2.0/feature/scm-ha.html | 539 +++++++++
docs/1.2.0/feature/scm-secure-ha.png | Bin 0 -> 46757 bytes
docs/1.2.0/feature/topology.html | 470 ++++++++
docs/1.2.0/fonts/glyphicons-halflings-regular.eot | Bin 0 -> 20127 bytes
docs/1.2.0/fonts/glyphicons-halflings-regular.svg | 288 +++++
docs/1.2.0/fonts/glyphicons-halflings-regular.ttf | Bin 0 -> 45404 bytes
docs/1.2.0/fonts/glyphicons-halflings-regular.woff | Bin 0 -> 23424 bytes
.../1.2.0/fonts/glyphicons-halflings-regular.woff2 | Bin 0 -> 18028 bytes
docs/1.2.0/index.html | 415 +++++++
docs/1.2.0/index.xml | 634 ++++++++++
docs/1.2.0/interface.html | 572 +++++++++
docs/1.2.0/interface/CSI.png | Bin 0 -> 27210 bytes
docs/1.2.0/interface/cli.html | 547 +++++++++
docs/1.2.0/interface/csi.html | 456 ++++++++
docs/1.2.0/interface/index.xml | 75 ++
docs/1.2.0/interface/javaapi.html | 515 ++++++++
docs/1.2.0/interface/o3fs.html | 479 ++++++++
docs/1.2.0/interface/ofs.html | 537 +++++++++
docs/1.2.0/interface/pvtest-webui.png | Bin 0 -> 13830 bytes
docs/1.2.0/interface/reconapi.html | 958 +++++++++++++++
docs/1.2.0/interface/s3.html | 573 +++++++++
docs/1.2.0/js/bootstrap.min.js | 6 +
docs/1.2.0/js/jquery-3.5.1.min.js | 2 +
docs/1.2.0/js/ozonedoc.js | 23 +
docs/1.2.0/ozone-logo-monochrome.svg | 195 ++++
docs/1.2.0/ozone-logo-small.png | Bin 0 -> 40237 bytes
docs/1.2.0/ozone-usage.png | Bin 0 -> 120495 bytes
docs/1.2.0/recipe.html | 486 ++++++++
docs/1.2.0/recipe/botoclient.html | 567 +++++++++
docs/1.2.0/recipe/index.xml | 38 +
docs/1.2.0/recipe/prometheus-key-allocate.png | Bin 0 -> 51155 bytes
docs/1.2.0/recipe/prometheus.html | 484 ++++++++
docs/1.2.0/recipe/prometheus.png | Bin 0 -> 38962 bytes
docs/1.2.0/recipe/sparkozonefsk8s.html | 512 ++++++++
docs/1.2.0/security.html | 621 ++++++++++
docs/1.2.0/security/gdpr.html | 463 ++++++++
docs/1.2.0/security/index.xml | 82 ++
docs/1.2.0/security/secureozone.html | 574 +++++++++
docs/1.2.0/security/securingdatanodes.html | 479 ++++++++
docs/1.2.0/security/securingozonehttp.html | 695 +++++++++++
docs/1.2.0/security/securings3.html | 451 +++++++
docs/1.2.0/security/securingtde.html | 503 ++++++++
docs/1.2.0/security/securityacls.html | 478 ++++++++
docs/1.2.0/security/securitywithranger.html | 528 +++++++++
docs/1.2.0/sitemap.xml | 18 +
docs/1.2.0/slides/README.md | 25 +
docs/1.2.0/slides/ozone-slides.drawio | 1 +
docs/1.2.0/slides/ozone.odp | Bin 0 -> 2034839 bytes
docs/1.2.0/slides/ozone.pdf | Bin 0 -> 1109486 bytes
docs/1.2.0/start.html | 597 ++++++++++
docs/1.2.0/start/docker.png | Bin 0 -> 7002 bytes
docs/1.2.0/start/fromsource.html | 456 ++++++++
docs/1.2.0/start/hadoop.png | Bin 0 -> 4923 bytes
docs/1.2.0/start/index.xml | 74 ++
docs/1.2.0/start/k8s.png | Bin 0 -> 6270 bytes
docs/1.2.0/start/kubernetes.html | 446 +++++++
docs/1.2.0/start/minikube.html | 460 ++++++++
docs/1.2.0/start/minikube.png | Bin 0 -> 5764 bytes
docs/1.2.0/start/onprem.html | 565 +++++++++
docs/1.2.0/start/runningviadocker.html | 459 ++++++++
docs/1.2.0/start/startfromdockerhub.html | 483 ++++++++
docs/1.2.0/tools.html | 532 +++++++++
docs/1.2.0/tools/admin.html | 429 +++++++
docs/1.2.0/tools/auditparser.html | 474 ++++++++
docs/1.2.0/tools/genconf.html | 425 +++++++
docs/1.2.0/tools/index.xml | 47 +
docs/1.2.0/tools/testtools.html | 590 ++++++++++
docs/1.2.0/zh/concept.html | 514 ++++++++
docs/1.2.0/zh/concept/ContainerMetadata.png | Bin 0 -> 98493 bytes
docs/1.2.0/zh/concept/Containers.png | Bin 0 -> 24775 bytes
docs/1.2.0/zh/concept/FunctionalOzone.png | Bin 0 -> 55334 bytes
docs/1.2.0/zh/concept/OzoneBlock.png | Bin 0 -> 4650 bytes
docs/1.2.0/zh/concept/OzoneManager-ReadPath.png | Bin 0 -> 81030 bytes
docs/1.2.0/zh/concept/OzoneManager-WritePath.png | Bin 0 -> 96696 bytes
docs/1.2.0/zh/concept/OzoneManager.png | Bin 0 -> 13327 bytes
docs/1.2.0/zh/concept/ReconHighLevelDesign.png | Bin 0 -> 239168 bytes
docs/1.2.0/zh/concept/ReconOmDesign.png | Bin 0 -> 162797 bytes
docs/1.2.0/zh/concept/ReconScmDesign.png | Bin 0 -> 181628 bytes
docs/1.2.0/zh/concept/StorageContainerManager.png | Bin 0 -> 13336 bytes
docs/1.2.0/zh/concept/containers.html | 416 +++++++
docs/1.2.0/zh/concept/datanodes.html | 399 +++++++
docs/1.2.0/zh/concept/index.xml | 65 ++
docs/1.2.0/zh/concept/overview.html | 398 +++++++
docs/1.2.0/zh/concept/ozoneBlockDiagram.png | Bin 0 -> 45218 bytes
docs/1.2.0/zh/concept/ozonemanager.html | 530 +++++++++
docs/1.2.0/zh/concept/recon.html | 533 +++++++++
docs/1.2.0/zh/concept/storagecontainermanager.html | 488 ++++++++
docs/1.2.0/zh/feature.html | 473 ++++++++
docs/1.2.0/zh/feature/HA-OM-doublebuffer.png | Bin 0 -> 77661 bytes
docs/1.2.0/zh/feature/HA-OM.png | Bin 0 -> 60888 bytes
docs/1.2.0/zh/feature/PrefixFSO-Delete.png | Bin 0 -> 253792 bytes
docs/1.2.0/zh/feature/PrefixFSO-Format.png | Bin 0 -> 270550 bytes
docs/1.2.0/zh/feature/PrefixFSO-Rename.png | Bin 0 -> 227851 bytes
docs/1.2.0/zh/feature/index.xml | 46 +
docs/1.2.0/zh/feature/om-ha.html | 439 +++++++
docs/1.2.0/zh/feature/quota.html | 447 +++++++
docs/1.2.0/zh/feature/recon.html | 387 ++++++
docs/1.2.0/zh/feature/scm-ha.html | 466 ++++++++
docs/1.2.0/zh/feature/scm-secure-ha.png | Bin 0 -> 46757 bytes
docs/1.2.0/zh/index.html | 378 ++++++
docs/1.2.0/zh/index.xml | 335 ++++++
docs/1.2.0/zh/interface.html | 491 ++++++++
docs/1.2.0/zh/interface/CSI.png | Bin 0 -> 27210 bytes
docs/1.2.0/zh/interface/csi.html | 423 +++++++
docs/1.2.0/zh/interface/index.xml | 55 +
docs/1.2.0/zh/interface/javaapi.html | 474 ++++++++
docs/1.2.0/zh/interface/o3fs.html | 457 ++++++++
docs/1.2.0/zh/interface/pvtest-webui.png | Bin 0 -> 13830 bytes
docs/1.2.0/zh/interface/reconapi.html | 715 ++++++++++++
docs/1.2.0/zh/interface/s3.html | 524 +++++++++
docs/1.2.0/zh/recipe.html | 428 +++++++
docs/1.2.0/zh/recipe/index.xml | 29 +
docs/1.2.0/zh/recipe/prometheus-key-allocate.png | Bin 0 -> 51155 bytes
docs/1.2.0/zh/recipe/prometheus.html | 446 +++++++
docs/1.2.0/zh/recipe/prometheus.png | Bin 0 -> 38962 bytes
docs/1.2.0/zh/recipe/sparkozonefsk8s.html | 479 ++++++++
docs/1.2.0/zh/security.html | 558 +++++++++
docs/1.2.0/zh/security/gdpr.html | 395 +++++++
docs/1.2.0/zh/security/index.xml | 73 ++
docs/1.2.0/zh/security/secureozone.html | 517 ++++++++
docs/1.2.0/zh/security/securingdatanodes.html | 423 +++++++
docs/1.2.0/zh/security/securings3.html | 404 +++++++
docs/1.2.0/zh/security/securingtde.html | 416 +++++++
docs/1.2.0/zh/security/securityacls.html | 422 +++++++
docs/1.2.0/zh/security/securitywithranger.html | 486 ++++++++
docs/1.2.0/zh/sitemap.xml | 538 +++++++++
docs/1.2.0/zh/start.html | 557 +++++++++
docs/1.2.0/zh/start/docker.png | Bin 0 -> 7002 bytes
docs/1.2.0/zh/start/fromsource.html | 414 +++++++
docs/1.2.0/zh/start/hadoop.png | Bin 0 -> 4923 bytes
docs/1.2.0/zh/start/index.xml | 92 ++
docs/1.2.0/zh/start/k8s.png | Bin 0 -> 6270 bytes
docs/1.2.0/zh/start/kubernetes.html | 410 +++++++
docs/1.2.0/zh/start/minikube.html | 424 +++++++
docs/1.2.0/zh/start/minikube.png | Bin 0 -> 5764 bytes
docs/1.2.0/zh/start/onprem.html | 504 ++++++++
docs/1.2.0/zh/start/runningviadocker.html | 415 +++++++
docs/1.2.0/zh/start/startfromdockerhub.html | 431 +++++++
docs/1.2.0/zh/tools.html | 470 ++++++++
docs/1.2.0/zh/tools/auditparser.html | 436 +++++++
docs/1.2.0/zh/tools/genconf.html | 388 ++++++
docs/1.2.0/zh/tools/index.xml | 38 +
docs/1.2.0/zh/tools/testtools.html | 555 +++++++++
docs/current | 2 +-
218 files changed, 61602 insertions(+), 1 deletion(-)
diff --git a/docs/1.2.0/NOTES.md b/docs/1.2.0/NOTES.md
new file mode 100644
index 0000000..b90edcf
--- /dev/null
+++ b/docs/1.2.0/NOTES.md
@@ -0,0 +1,20 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+The source of Ozone logo is available here:
+
+https://gitbox.apache.org/repos/asf?p=hadoop-site.git;a=tree;f=ozone/static;hb=refs/heads/asf-site
diff --git a/docs/1.2.0/OzoneOverview.png b/docs/1.2.0/OzoneOverview.png
new file mode 100644
index 0000000..7e011d5
Binary files /dev/null and b/docs/1.2.0/OzoneOverview.png differ
diff --git a/docs/1.2.0/OzoneOverview.svg b/docs/1.2.0/OzoneOverview.svg
new file mode 100644
index 0000000..9d4660d
--- /dev/null
+++ b/docs/1.2.0/OzoneOverview.svg
@@ -0,0 +1,238 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<svg width="703px" height="465px" viewBox="0 0 703 465" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+ <!-- Generator: Sketch 44.1 (41455) - http://www.bohemiancoding.com/sketch -->
+ <title>Desktop HD</title>
+ <desc>Created with Sketch.</desc>
+ <defs>
+ <rect id="path-1" x="0" y="0" width="131" height="36" rx="8"></rect>
+ <rect id="path-2" x="0" y="0" width="131" height="36" rx="8"></rect>
+ <rect id="path-3" x="9" y="304" width="437" height="144"></rect>
+ <mask id="mask-4" maskContentUnits="userSpaceOnUse" maskUnits="objectBoundingBox" x="0" y="0" width="437" height="144" fill="white">
+ <use xlink:href="#path-3"></use>
+ </mask>
+ <rect id="path-5" x="0" y="0" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-6" x="3.26727637" y="4.49286685" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-7" x="7.07909881" y="11.2321671" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-8" x="0" y="0" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-9" x="3.26727637" y="4.49286685" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-10" x="7.07909881" y="11.2321671" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-11" x="0" y="0" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-12" x="3.26727637" y="4.49286685" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-13" x="7.07909881" y="11.2321671" width="123.06741" height="40.4358016" rx="8"></rect>
+ <rect id="path-14" x="0" y="14.7446809" width="97" height="36.1914894"></rect>
+ <rect id="path-15" x="0" y="14.7446809" width="97" height="36.1914894"></rect>
+ <rect id="path-16" x="0" y="0" width="131" height="36" rx="8"></rect>
+ <rect id="path-17" x="0" y="0" width="131" height="36" rx="8"></rect>
+ <rect id="path-18" x="0.140758874" y="0" width="142.859241" height="35.1071084" rx="8"></rect>
+ <rect id="path-19" x="0" y="0" width="226" height="36" rx="8"></rect>
+ <rect id="path-20" x="6" y="4" width="226" height="36" rx="8"></rect>
+ <rect id="path-21" x="13" y="10" width="226" height="36" rx="8"></rect>
+ <rect id="path-22" x="0" y="0" width="226" height="36" rx="8"></rect>
+ <rect id="path-23" x="6" y="4" width="226" height="36" rx="8"></rect>
+ <rect id="path-24" x="13" y="10" width="226" height="36" rx="8"></rect>
+ <rect id="path-25" x="0" y="14.7446809" width="97" height="36.1914894"></rect>
+ </defs>
+ <g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
+ <path d="M84.5,51.5 L240.5,130.5" id="Line" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path id="Line-decoration-1" d="M240.5,130.5 L232.220366,122.944362 L229.50967,128.29713 L240.5,130.5 Z" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path d="M142.5,150.5 L177.5,150.5" id="Line" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path id="Line-decoration-1" d="M177.5,150.5 L166.7,147.5 L166.7,153.5 L177.5,150.5 Z" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <g id="Desktop-HD">
+ <g id="Client" transform="translate(176.000000, 132.000000)">
+ <g id="Rectangle">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-1"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="130" height="35" rx="8"></rect>
+ </g>
+ <text id="Ozone-Client" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="24.2107393" y="24">Ozone Client</tspan>
+ </text>
+ </g>
+ <g id="Handler" transform="translate(35.000000, 18.000000)">
+ <g id="Rectangle-2">
+ <use fill="#E4D6F8" fill-rule="evenodd" xlink:href="#path-2"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="130" height="35" rx="8"></rect>
+ </g>
+ <text id="Rest-Handler" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="22.3208008" y="22">Rest Handler</tspan>
+ </text>
+ </g>
+ <use id="Rectangle-4" stroke="#979797" mask="url(#mask-4)" stroke-width="2" fill="#FFFFFF" stroke-dasharray="1,3,1,3" xlink:href="#path-3"></use>
+ <g id="Ratis" transform="translate(315.000000, 378.000000)">
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-5"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-6"></use>
+ <rect stroke="#000000" stroke-width="1" x="3.76727637" y="4.99286685" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-7"></use>
+ <rect stroke="#000000" stroke-width="1" x="7.57909881" y="11.7321671" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <text font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="51.3828125" y="35.9642655">Ratis</tspan>
+ </text>
+ </g>
+ <g id="Ratis" transform="translate(166.000000, 378.000000)">
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-8"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-9"></use>
+ <rect stroke="#000000" stroke-width="1" x="3.76727637" y="4.99286685" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-10"></use>
+ <rect stroke="#000000" stroke-width="1" x="7.57909881" y="11.7321671" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <text font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="51.3828125" y="35.9642655">Ratis</tspan>
+ </text>
+ </g>
+ <g id="Ratis" transform="translate(10.000000, 378.000000)">
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-11"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-12"></use>
+ <rect stroke="#000000" stroke-width="1" x="3.76727637" y="4.99286685" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-13"></use>
+ <rect stroke="#000000" stroke-width="1" x="7.57909881" y="11.7321671" width="122.06741" height="39.4358016" rx="8"></rect>
+ </g>
+ <text font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="51.3828125" y="35.9642655">Ratis</tspan>
+ </text>
+ </g>
+ <path d="M240.5,168.5 L240.5,311.5" id="Line" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path id="Line-decoration-1" d="M240.5,311.5 L243.5,300.7 L237.5,300.7 L240.5,311.5 Z" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path d="M243.5,54.5 L243.5,131.5" id="Line" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path id="Line-decoration-1" d="M243.5,131.5 L246.5,120.7 L240.5,120.7 L243.5,131.5 Z" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <g id="Container" transform="translate(328.000000, 313.000000)">
+ <g id="Rectangle-5">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-14"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="15.2446809" width="96" height="35.1914894"></rect>
+ </g>
+ <ellipse id="Oval" stroke="#000000" fill="#C6D4F9" cx="48.5" cy="11.393617" rx="48.5" ry="11.393617"></ellipse>
+ <ellipse id="Oval" stroke="#000000" fill="#C6D4F9" cx="48.5" cy="51.606383" rx="48.5" ry="11.393617"></ellipse>
+ <text font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="14.3310547" y="37">Container</tspan>
+ </text>
+ </g>
+ <g id="Container" transform="translate(176.000000, 312.000000)">
+ <g id="Rectangle-5">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-15"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="15.2446809" width="96" height="35.1914894"></rect>
+ </g>
+ <ellipse id="Oval" stroke="#000000" fill="#C6D4F9" cx="48.5" cy="11.393617" rx="48.5" ry="11.393617"></ellipse>
+ <ellipse id="Oval" stroke="#000000" fill="#C6D4F9" cx="48.5" cy="51.606383" rx="48.5" ry="11.393617"></ellipse>
+ <text font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="14.3310547" y="37">Container</tspan>
+ </text>
+ </g>
+ <g id="FileSystem" transform="translate(11.000000, 133.000000)">
+ <g id="Rectangle-2">
+ <use fill="#7ED321" fill-rule="evenodd" xlink:href="#path-16"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="130" height="35" rx="8"></rect>
+ </g>
+ <text id="Ozone-File-System" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="3.25878906" y="22">Ozone File System</tspan>
+ </text>
+ </g>
+ <g id="CLI" transform="translate(179.000000, 18.000000)">
+ <g id="Rectangle-2">
+ <use fill="#E4D6F8" fill-rule="evenodd" xlink:href="#path-17"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="130" height="35" rx="8"></rect>
+ </g>
+ <text id="Ozone-CLI" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="26.3896484" y="23">Ozone CLI</tspan>
+ </text>
+ </g>
+ <path d="M333.336323,48.7787611 L248.494492,130.227891" id="Line" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path id="Line-decoration-1" d="M248.494492,130.227891 L258.363039,124.91265 L254.207822,120.584351 L248.494492,130.227891 Z" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <g id="Corona" transform="translate(325.000000, 17.000000)">
+ <g id="Rectangle-2">
+ <use fill="#E4D6F8" fill-rule="evenodd" xlink:href="#path-18"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.640758874" y="0.5" width="141.859241" height="34.1071084" rx="8"></rect>
+ </g>
+ <text id="Freon" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="50.7326544" y="22.8128606">Freon</tspan>
+ </text>
+ </g>
+ <path d="M307.5,148.5 L433.5,148.5" id="Line" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path id="Line-decoration-1" d="M433.5,148.5 L422.7,145.5 L422.7,151.5 L433.5,148.5 Z" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path d="M4,232 L699,232" id="Line" stroke="#000000" stroke-width="2" stroke-linecap="square" stroke-dasharray="5,2,5"></path>
+ <g id="OM" transform="translate(432.000000, 132.000000)">
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-19"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="225" height="35" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-20"></use>
+ <rect stroke="#000000" stroke-width="1" x="6.5" y="4.5" width="225" height="35" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-21"></use>
+ <rect stroke="#000000" stroke-width="1" x="13.5" y="10.5" width="225" height="35" rx="8"></rect>
+ </g>
+ <text id="Ozone-Manager" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="67.3793945" y="32">Ozone Manager</tspan>
+ </text>
+ </g>
+ <g id="SCM" transform="translate(450.000000, 281.000000)">
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-22"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="0.5" width="225" height="35" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-23"></use>
+ <rect stroke="#000000" stroke-width="1" x="6.5" y="4.5" width="225" height="35" rx="8"></rect>
+ </g>
+ <g id="Rectangle-3">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-24"></use>
+ <rect stroke="#000000" stroke-width="1" x="13.5" y="10.5" width="225" height="35" rx="8"></rect>
+ </g>
+ <text id="Storage-Container-Manager" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="28.0932617" y="32">Storage Container Manager</tspan>
+ </text>
+ </g>
+ <path d="M534.5,178.5 L534.5,283.5" id="Line" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <path id="Line-decoration-1" d="M534.5,283.5 L537.5,272.7 L531.5,272.7 L534.5,283.5 Z" stroke="#000000" fill="#000000" stroke-linecap="square"></path>
+ <text id="Datanodes" font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="196.213867" y="462">Datanodes</tspan>
+ </text>
+ <g id="Container" transform="translate(15.000000, 311.000000)">
+ <g id="Rectangle-5">
+ <use fill="#C6D4F9" fill-rule="evenodd" xlink:href="#path-25"></use>
+ <rect stroke="#000000" stroke-width="1" x="0.5" y="15.2446809" width="96" height="35.1914894"></rect>
+ </g>
+ <ellipse id="Oval" stroke="#000000" fill="#C6D4F9" cx="48.5" cy="11.393617" rx="48.5" ry="11.393617"></ellipse>
+ <ellipse id="Oval" stroke="#000000" fill="#C6D4F9" cx="48.5" cy="51.606383" rx="48.5" ry="11.393617"></ellipse>
+ <text font-family="Helvetica-Bold, Helvetica" font-size="14" font-weight="bold" fill="#000000">
+ <tspan x="14.3310547" y="37">Container</tspan>
+ </text>
+ </g>
+ <path d="M84.5,137.5 L86,139" id="Line" stroke="#979797" stroke-linecap="square"></path>
+ <text id="Hadoop-Distributed-D" font-family="Helvetica-Bold, Helvetica" font-size="20" font-weight="bold" fill="#000000">
+ <tspan x="205.433594" y="230">Hadoop Distributed Data Store</tspan>
+ </text>
+ </g>
+ </g>
+</svg>
\ No newline at end of file
diff --git a/docs/1.2.0/SCMBlockDiagram.png b/docs/1.2.0/SCMBlockDiagram.png
new file mode 100644
index 0000000..04d27ad
Binary files /dev/null and b/docs/1.2.0/SCMBlockDiagram.png differ
diff --git a/docs/1.2.0/concept.html b/docs/1.2.0/concept.html
new file mode 100644
index 0000000..4b7f24d
--- /dev/null
+++ b/docs/1.2.0/concept.html
@@ -0,0 +1,553 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="./css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="./css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="./index.html" class="navbar-left ozone-logo">
+ <img src="./ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="./index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+<div class="wrapper">
+<div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="./index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="./start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="active">
+ <a href="./concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="./feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="./interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="./security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="./tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="./recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="./design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+
+
+
+<div class="pull-right">
+
+
+
+
+
+ <a href="./zh/concept.html"><span class="label label-success">中文</span></a>
+
+
+</div>
+
+ <h1>Architecture</h1>
+ </div>
+
+ <div class="col-md-9">
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+
+<div class="jumbotron jumbotron-fluid">
+ <div class="container">
+ <h3 class="display-4">Ozone Architecture </h3>
+ <p class="lead">
+
+
+Ozone's architectural elements are explained in the following pages. The
+metadata layer, data layer, protocol bus, replication layer and Recon are
+discussed in the following pages. These concepts are useful if you want to
+understand how ozone works in depth.
+
+
+ </p>
+ </div>
+</div>
+
+
+
+
+
+
+
+ <div class="row">
+
+ <div class="col-sm-6">
+ <div class="card">
+ <div class="card-body">
+ <h2 class="card-title">
+
+ Overview
+ </h2>
+ <p class="card-text">Ozone’s overview and components that make up Ozone.</p>
+ <a href="./concept/overview.html"
+ class=" btn btn-primary btn-lg">Overview</a>
+ </div>
+ </div>
+ </div>
+
+
+
+
+
+
+ <div class="col-sm-6">
+ <div class="card">
+ <div class="card-body">
+ <h2 class="card-title">
+
+ Ozone Manager
+ </h2>
+ <p class="card-text">Ozone Manager is the principal name space service of Ozone. OM manages the life cycle of volumes, buckets and Keys.</p>
+ <a href="./concept/ozonemanager.html"
+ class=" btn btn-primary btn-lg">Ozone Manager</a>
+ </div>
+ </div>
+ </div>
+
+
+ </div>
+
+
+
+
+
+ <div class="row">
+
+ <div class="col-sm-6">
+ <div class="card">
+ <div class="card-body">
+ <h2 class="card-title">
+
+ Storage Container Manager
+ </h2>
+ <p class="card-text">Storage Container Manager or SCM is the core metadata service of Ozone. SCM provides a distributed block layer for Ozone.</p>
+ <a href="./concept/storagecontainermanager.html"
+ class=" btn btn-primary btn-lg">Storage Container Manager</a>
+ </div>
+ </div>
+ </div>
+
+
+
+
+
+
+ <div class="col-sm-6">
+ <div class="card">
+ <div class="card-body">
+ <h2 class="card-title">
+
+ Containers
+ </h2>
+ <p class="card-text">Description of the Containers, the replication unit of Ozone.</p>
+ <a href="./concept/containers.html"
+ class=" btn btn-primary btn-lg">Containers</a>
+ </div>
+ </div>
+ </div>
+
+
+ </div>
+
+
+
+
+
+ <div class="row">
+
+ <div class="col-sm-6">
+ <div class="card">
+ <div class="card-body">
+ <h2 class="card-title">
+
+ Datanodes
+ </h2>
+ <p class="card-text">Ozone supports Amazon’s Simple Storage Service (S3) protocol. In fact, You can use S3 clients and S3 SDK based applications without any modifications with Ozone.</p>
+ <a href="./concept/datanodes.html"
+ class=" btn btn-primary btn-lg">Datanodes</a>
+ </div>
+ </div>
+ </div>
+
+
+
+
+
+
+ <div class="col-sm-6">
+ <div class="card">
+ <div class="card-body">
+ <h2 class="card-title">
+
+ Recon
+ </h2>
+ <p class="card-text">Recon serves as a management and monitoring console for Ozone.</p>
+ <a href="./concept/recon.html"
+ class=" btn btn-primary btn-lg">Recon</a>
+ </div>
+ </div>
+ </div>
+
+
+ </div>
+
+
+
+ </div>
+ </div>
+ </div>
+</div>
+ <div class="push"></div>
+</div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: August 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/5ce6f0eab381389fd04c3130531b3ec626acbc65">5ce6f0eab</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="./js/jquery-3.5.1.min.js"></script>
+<script src="./js/ozonedoc.js"></script>
+<script src="./js/bootstrap.min.js"></script>
+
+
+</body>
+
+</html>
\ No newline at end of file
diff --git a/docs/1.2.0/concept/ContainerMetadata.png b/docs/1.2.0/concept/ContainerMetadata.png
new file mode 100644
index 0000000..48bd1c4
Binary files /dev/null and b/docs/1.2.0/concept/ContainerMetadata.png differ
diff --git a/docs/1.2.0/concept/Containers.png b/docs/1.2.0/concept/Containers.png
new file mode 100644
index 0000000..3d2df0f
Binary files /dev/null and b/docs/1.2.0/concept/Containers.png differ
diff --git a/docs/1.2.0/concept/FunctionalOzone.png b/docs/1.2.0/concept/FunctionalOzone.png
new file mode 100644
index 0000000..0bc75b5
Binary files /dev/null and b/docs/1.2.0/concept/FunctionalOzone.png differ
diff --git a/docs/1.2.0/concept/OzoneBlock.png b/docs/1.2.0/concept/OzoneBlock.png
new file mode 100644
index 0000000..9583bd5
Binary files /dev/null and b/docs/1.2.0/concept/OzoneBlock.png differ
diff --git a/docs/1.2.0/concept/OzoneManager-ReadPath.png b/docs/1.2.0/concept/OzoneManager-ReadPath.png
new file mode 100644
index 0000000..5e68f6f
Binary files /dev/null and b/docs/1.2.0/concept/OzoneManager-ReadPath.png differ
diff --git a/docs/1.2.0/concept/OzoneManager-WritePath.png b/docs/1.2.0/concept/OzoneManager-WritePath.png
new file mode 100644
index 0000000..924b61c
Binary files /dev/null and b/docs/1.2.0/concept/OzoneManager-WritePath.png differ
diff --git a/docs/1.2.0/concept/OzoneManager.png b/docs/1.2.0/concept/OzoneManager.png
new file mode 100644
index 0000000..f71bfac
Binary files /dev/null and b/docs/1.2.0/concept/OzoneManager.png differ
diff --git a/docs/1.2.0/concept/ReconHighLevelDesign.png b/docs/1.2.0/concept/ReconHighLevelDesign.png
new file mode 100644
index 0000000..3bd6443
Binary files /dev/null and b/docs/1.2.0/concept/ReconHighLevelDesign.png differ
diff --git a/docs/1.2.0/concept/ReconOmDesign.png b/docs/1.2.0/concept/ReconOmDesign.png
new file mode 100644
index 0000000..20ea6a3
Binary files /dev/null and b/docs/1.2.0/concept/ReconOmDesign.png differ
diff --git a/docs/1.2.0/concept/ReconScmDesign.png b/docs/1.2.0/concept/ReconScmDesign.png
new file mode 100644
index 0000000..32d07e0
Binary files /dev/null and b/docs/1.2.0/concept/ReconScmDesign.png differ
diff --git a/docs/1.2.0/concept/StorageContainerManager.png b/docs/1.2.0/concept/StorageContainerManager.png
new file mode 100644
index 0000000..605c48c
Binary files /dev/null and b/docs/1.2.0/concept/StorageContainerManager.png differ
diff --git a/docs/1.2.0/concept/containers.html b/docs/1.2.0/concept/containers.html
new file mode 100644
index 0000000..d72acdf
--- /dev/null
+++ b/docs/1.2.0/concept/containers.html
@@ -0,0 +1,452 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+ <div class="wrapper">
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="active">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
+
+
+
+ <div class="col-md-9">
+ <nav aria-label="breadcrumb">
+ <ol class="breadcrumb">
+ <li class="breadcrumb-item"><a href="../index.html">Home</a></li>
+ <li class="breadcrumb-item" aria-current="page"><a href="../concept.html">Architecture</a></li>
+ <li class="breadcrumb-item active" aria-current="page">Containers</li>
+ </ol>
+ </nav>
+
+
+
+<div class="pull-right">
+
+
+
+
+
+ <a href="../zh/concept/containers.html"><span class="label label-success">中文</span></a>
+
+
+</div>
+
+
+ <div class="col-md-9">
+ <h1>Containers</h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<p>Containers are the fundamental replication unit of Ozone/HDDS, they are managed by the Storage Container Manager (SCM) service.</p>
+<p>Containers are big binary units (5Gb by default) which can contain multiple blocks:</p>
+<p><img src="Containers.png" alt="Containers"></p>
+<p>Blocks are local information and not managed by SCM. Therefore even if billions of small files are created in the system (which means billions of blocks are created), only of the status of the containers will be reported by the Datanodes and containers will be replicated.</p>
+<p>When Ozone Manager requests a new Block allocation from the SCM, SCM will identify the suitable container and generate a block id which contains <code>ContainerId</code> + <code>LocalId</code>. Client will connect to the Datanode which stores the Container, and datanode can manage the separated block based on the <code>LocalId</code>.</p>
+<h2 id="open-vs-closed-containers">Open vs. Closed containers</h2>
+<p>When a container is created it starts in an OPEN state. When it’s full (~5GB data is written), container will be closed and becomes a CLOSED container.</p>
+<p>The fundamental differences between OPEN and CLOSED containers:</p>
+<table>
+<thead>
+<tr>
+<th>OPEN</th>
+<th>CLOSED</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>mutable</td>
+<td>immutable</td>
+</tr>
+<tr>
+<td>replicated with RAFT (Ratis)</td>
+<td>Replicated with async container copy</td>
+</tr>
+<tr>
+<td>Raft leader is used to READ / WRITE</td>
+<td>All the nodes can be used to READ</td>
+</tr>
+</tbody>
+</table>
+
+
+
+ <a class="btn btn-success btn-lg" href="../concept/datanodes.html">Next >></a>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="push"></div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: August 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/5ce6f0eab381389fd04c3130531b3ec626acbc65">5ce6f0eab</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+</body>
+
+</html>
\ No newline at end of file
diff --git a/docs/1.2.0/concept/datanodes.html b/docs/1.2.0/concept/datanodes.html
new file mode 100644
index 0000000..3997f01
--- /dev/null
+++ b/docs/1.2.0/concept/datanodes.html
@@ -0,0 +1,461 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+ <div class="wrapper">
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="active">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
+
+
+
+ <div class="col-md-9">
+ <nav aria-label="breadcrumb">
+ <ol class="breadcrumb">
+ <li class="breadcrumb-item"><a href="../index.html">Home</a></li>
+ <li class="breadcrumb-item" aria-current="page"><a href="../concept.html">Architecture</a></li>
+ <li class="breadcrumb-item active" aria-current="page">Datanodes</li>
+ </ol>
+ </nav>
+
+
+
+<div class="pull-right">
+
+
+
+
+
+ <a href="../zh/concept/datanodes.html"><span class="label label-success">中文</span></a>
+
+
+</div>
+
+
+ <div class="col-md-9">
+ <h1>Datanodes</h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<p>Datanodes are the worker bees of Ozone. All data is stored on data nodes.
+Clients write data in terms of blocks. Datanode aggregates these blocks into
+a storage container. A storage container is the data streams and metadata
+about the blocks written by the clients.</p>
+<h2 id="storage-containers">Storage Containers</h2>
+<p><img src="ContainerMetadata.png" alt="FunctionalOzone"></p>
+<p>A storage container is a self-contained super block. It has a list of Ozone
+blocks that reside inside it, as well as on-disk files which contain the
+actual data streams. This is the default Storage container format. From
+Ozone’s perspective, container is a protocol spec, actual storage layouts
+does not matter. In other words, it is trivial to extend or bring new
+container layouts. Hence this should be treated as a reference implementation
+of containers under Ozone.</p>
+<h2 id="understanding-ozone-blocks-and-containers">Understanding Ozone Blocks and Containers</h2>
+<p>When a client wants to read a key from Ozone, the client sends the name of
+the key to the Ozone Manager. Ozone manager returns the list of Ozone blocks
+that make up that key.</p>
+<p>An Ozone block contains the container ID and a local ID. The figure below
+shows the logical layout out of Ozone block.</p>
+<p><img src="OzoneBlock.png" alt="OzoneBlock"></p>
+<p>The container ID lets the clients discover the location of the container. The
+authoritative information about where a container is located is with the
+Storage Container Manager (SCM). In most cases, the container location will be
+cached by Ozone Manager and will be returned along with the Ozone blocks.</p>
+<p>Once the client is able to locate the container, that is, understand which
+data nodes contain this container, the client will connect to the datanode
+and read the data stream specified by <em>Container ID:Local ID</em>. In other
+words, the local ID serves as index into the container which describes what
+data stream we want to read from.</p>
+<h3 id="discovering-the-container-locations">Discovering the Container Locations</h3>
+<p>How does SCM know where the containers are located ? This is very similar to
+what HDFS does; the data nodes regularly send container reports like block
+reports. Container reports are far more concise than block reports. For
+example, an Ozone deployment with a 196 TB data node will have around 40
+thousand containers. Compare that with HDFS block count of million and half
+blocks that get reported. That is a 40x reduction in the block reports.</p>
+<p>This extra indirection helps tremendously with scaling Ozone. SCM has far
+less block data to process and the namespace service (Ozone Manager) as a
+different service are critical to scaling Ozone.</p>
+
+
+
+ <a class="btn btn-success btn-lg" href="../concept/recon.html">Next >></a>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="push"></div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: August 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/5ce6f0eab381389fd04c3130531b3ec626acbc65">5ce6f0eab</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+</body>
+
+</html>
\ No newline at end of file
diff --git a/docs/1.2.0/concept/index.xml b/docs/1.2.0/concept/index.xml
new file mode 100644
index 0000000..789911a
--- /dev/null
+++ b/docs/1.2.0/concept/index.xml
@@ -0,0 +1,65 @@
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+ <channel>
+ <title>Architecture on Ozone</title>
+ <link>/concept.html</link>
+ <description>Recent content in Architecture on Ozone</description>
+ <generator>Hugo -- gohugo.io</generator>
+ <language>en-us</language>
+ <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate><atom:link href="/concept/index.xml" rel="self" type="application/rss+xml" />
+ <item>
+ <title>Overview</title>
+ <link>/concept/overview.html</link>
+ <pubDate>Tue, 10 Oct 2017 00:00:00 +0000</pubDate>
+
+ <guid>/concept/overview.html</guid>
+ <description>Ozone&rsquo;s overview and components that make up Ozone.</description>
+ </item>
+
+ <item>
+ <title>Ozone Manager</title>
+ <link>/concept/ozonemanager.html</link>
+ <pubDate>Thu, 14 Sep 2017 00:00:00 +0000</pubDate>
+
+ <guid>/concept/ozonemanager.html</guid>
+ <description>Ozone Manager is the principal name space service of Ozone. OM manages the life cycle of volumes, buckets and Keys.</description>
+ </item>
+
+ <item>
+ <title>Storage Container Manager</title>
+ <link>/concept/storagecontainermanager.html</link>
+ <pubDate>Thu, 14 Sep 2017 00:00:00 +0000</pubDate>
+
+ <guid>/concept/storagecontainermanager.html</guid>
+ <description>Storage Container Manager or SCM is the core metadata service of Ozone. SCM provides a distributed block layer for Ozone.</description>
+ </item>
+
+ <item>
+ <title>Containers</title>
+ <link>/concept/containers.html</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+
+ <guid>/concept/containers.html</guid>
+ <description>Description of the Containers, the replication unit of Ozone.</description>
+ </item>
+
+ <item>
+ <title>Datanodes</title>
+ <link>/concept/datanodes.html</link>
+ <pubDate>Thu, 14 Sep 2017 00:00:00 +0000</pubDate>
+
+ <guid>/concept/datanodes.html</guid>
+ <description>Ozone supports Amazon&rsquo;s Simple Storage Service (S3) protocol. In fact, You can use S3 clients and S3 SDK based applications without any modifications with Ozone.</description>
+ </item>
+
+ <item>
+ <title>Recon</title>
+ <link>/concept/recon.html</link>
+ <pubDate>Tue, 27 Oct 2020 00:00:00 +0000</pubDate>
+
+ <guid>/concept/recon.html</guid>
+ <description>Recon serves as a management and monitoring console for Ozone.</description>
+ </item>
+
+ </channel>
+</rss>
diff --git a/docs/1.2.0/concept/overview.html b/docs/1.2.0/concept/overview.html
new file mode 100644
index 0000000..669976a
--- /dev/null
+++ b/docs/1.2.0/concept/overview.html
@@ -0,0 +1,461 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+ <div class="wrapper">
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="active">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
+
+
+
+ <div class="col-md-9">
+ <nav aria-label="breadcrumb">
+ <ol class="breadcrumb">
+ <li class="breadcrumb-item"><a href="../index.html">Home</a></li>
+ <li class="breadcrumb-item" aria-current="page"><a href="../concept.html">Architecture</a></li>
+ <li class="breadcrumb-item active" aria-current="page">Overview</li>
+ </ol>
+ </nav>
+
+
+
+<div class="pull-right">
+
+
+
+
+
+ <a href="../zh/concept/overview.html"><span class="label label-success">中文</span></a>
+
+
+</div>
+
+
+ <div class="col-md-9">
+ <h1>Overview</h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<p>Ozone is a redundant, distributed object store optimized for Big data
+workloads. The primary design point of ozone is scalability, and it aims to
+scale to billions of objects.</p>
+<p>Ozone separates namespace management and block space management; this helps
+ozone to scale much better. The namespace is managed by a daemon called
+<a href="../concept/ozonemanager.html">Ozone Manager </a> (OM), and block space is
+managed by <a href="../concept/storagecontainermanager.html">Storage Container Manager</a> (SCM).</p>
+<p>Ozone consists of volumes, buckets, and keys.
+A volume is similar to a home directory in the ozone world.
+Only an administrator can create it.</p>
+<p>Volumes are used to store buckets.
+Once a volume is created users can create as many buckets as needed.
+Ozone stores data as keys which live inside these buckets.</p>
+<p>Ozone namespace is composed of many storage volumes.
+Storage volumes are also used as the basis for storage accounting.</p>
+<p>The block diagram shows the core components of Ozone.</p>
+<p><img src="ozoneBlockDiagram.png" alt="Architecture diagram"></p>
+<p>The Ozone Manager is the name space manager, Storage Container Manager
+manages the physical and data layer and Recon is the management interface for
+Ozone.</p>
+<h2 id="different-perspectives">Different Perspectives</h2>
+<p><img src="FunctionalOzone.png" alt="FunctionalOzone"></p>
+<p>Any distributed system can be viewed from different perspectives. One way to
+look at Ozone is to imagine it as Ozone Manager as a name space service built on
+top of HDDS, a distributed block store.</p>
+<p>Another way to visualize Ozone is to look at the functional layers; we have a
+metadata data management layer, composed of Ozone Manager and Storage
+Container Manager.</p>
+<p>We have a data storage layer, which is basically the data nodes and they are
+managed by SCM.</p>
+<p>The replication layer, provided by Ratis is used to replicate metadata (OM and SCM)
+and also used for consistency when data is modified at the
+data nodes.</p>
+<p>We have a management server called Recon, that talks to all other components
+of Ozone and provides a unified management API and UX for Ozone.</p>
+<p>We have a protocol bus that allows Ozone to be extended via other
+protocols. We currently only have S3 protocol support built via Protocol bus.
+Protocol Bus provides a generic notion that you can implement new file system
+or object store protocols that call into O3 Native protocol.</p>
+
+
+
+ <a class="btn btn-success btn-lg" href="../concept/ozonemanager.html">Next >></a>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="push"></div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: August 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/5ce6f0eab381389fd04c3130531b3ec626acbc65">5ce6f0eab</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+</body>
+
+</html>
\ No newline at end of file
diff --git a/docs/1.2.0/concept/ozoneBlockDiagram.png b/docs/1.2.0/concept/ozoneBlockDiagram.png
new file mode 100644
index 0000000..7fb738f
Binary files /dev/null and b/docs/1.2.0/concept/ozoneBlockDiagram.png differ
diff --git a/docs/1.2.0/concept/ozonemanager.html b/docs/1.2.0/concept/ozonemanager.html
new file mode 100644
index 0000000..4dd37fe
--- /dev/null
+++ b/docs/1.2.0/concept/ozonemanager.html
@@ -0,0 +1,578 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+ <div class="wrapper">
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="active">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
+
+
+
+ <div class="col-md-9">
+ <nav aria-label="breadcrumb">
+ <ol class="breadcrumb">
+ <li class="breadcrumb-item"><a href="../index.html">Home</a></li>
+ <li class="breadcrumb-item" aria-current="page"><a href="../concept.html">Architecture</a></li>
+ <li class="breadcrumb-item active" aria-current="page">Ozone Manager</li>
+ </ol>
+ </nav>
+
+
+
+<div class="pull-right">
+
+
+
+
+
+ <a href="../zh/concept/ozonemanager.html"><span class="label label-success">中文</span></a>
+
+
+</div>
+
+
+ <div class="col-md-9">
+ <h1>Ozone Manager</h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<p><img src="OzoneManager.png" alt="Ozone Manager"></p>
+<p>Ozone Manager (OM) is the namespace manager for Ozone.</p>
+<p>This means that when you want to write some data, you ask Ozone
+Manager for a block and Ozone Manager gives you a block and remembers that
+information. When you want to read that file back, you need to find the
+address of the block and Ozone Manager returns it you.</p>
+<p>Ozone Manager also allows users to organize keys under a volume and bucket.
+Volumes and buckets are part of the namespace and managed by Ozone Manager.</p>
+<p>Each ozone volume is the root of an independent namespace under OM.
+This is very different from HDFS which provides a single rooted file system.</p>
+<p>Ozone’s namespace is a collection of volumes or is a forest instead of a
+single rooted tree as in HDFS. This property makes it easy to deploy multiple
+OMs for scaling(pending future development).</p>
+<h2 id="ozone-manager-metadata">Ozone Manager Metadata</h2>
+<p>OM maintains a list of volumes, buckets, and keys.
+For each user, it maintains a list of volumes.
+For each volume, the list of buckets and for each bucket the list of keys.</p>
+<p>Ozone Manager will use Apache Ratis(A Raft protocol implementation) to
+replicate Ozone Manager state. This will ensure High Availability for Ozone.</p>
+<h2 id="ozone-manager-and-storage-container-manager">Ozone Manager and Storage Container Manager</h2>
+<p>The relationship between Ozone Manager and Storage Container Manager is best
+understood if we trace what happens during a key write and key read.</p>
+<h3 id="key-write">Key Write</h3>
+<p><img src="OzoneManager-WritePath.png" alt="Write Path"></p>
+<ul>
+<li>
+<p>To write a key to Ozone, a client tells Ozone manager that it would like to
+write a key into a bucket that lives inside a specific volume. Once Ozone
+Manager determines that you are allowed to write a key to the specified bucket,
+OM needs to allocate a block for the client to write data.</p>
+</li>
+<li>
+<p>To allocate a block, Ozone Manager sends a request to Storage Container
+Manager (SCM); SCM is the manager of data nodes. SCM picks three data nodes
+into which client can write data. SCM allocates the block and returns the
+block ID to Ozone Manager.</p>
+</li>
+<li>
+<p>Ozone manager records this block information in its metadata and returns the
+block and a block token (a security permission to write data to the block)
+to the client.</p>
+</li>
+<li>
+<p>The client uses the block token to prove that it is allowed to write data to
+the block and writes data to the data node.</p>
+</li>
+<li>
+<p>Once the write is complete on the data node, the client will update the block
+information on Ozone manager.</p>
+</li>
+</ul>
+<h3 id="key-reads">Key Reads</h3>
+<p><img src="OzoneManager-ReadPath.png" alt="Read Path"></p>
+<ul>
+<li>Key reads are simpler, the client requests the block list from the Ozone
+Manager</li>
+<li>Ozone manager will return the block list and block tokens which
+allows the client to read the data from data nodes.</li>
+<li>Client connects to the data node and presents the block token and reads
+the data from the data node.</li>
+</ul>
+<h2 id="main-components-of-the-ozone-manager">Main components of the Ozone Manager</h2>
+<p>For a detailed view of Ozone Manager this section gives a quick overview about the provided network services and the stored persisted data.</p>
+<h3 id="network-services-provided-by-ozone-manager">Network services provided by Ozone Manager:</h3>
+<p>Ozone provides a network service for the client and for administration commands. The main service calls</p>
+<ul>
+<li>Key, Bucket, Volume / CRUD</li>
+<li>Multipart upload (Initiate, Complete…)
+<ul>
+<li>Supports upload of huge files in multiple steps</li>
+</ul>
+</li>
+<li>FS related calls (optimized for hierarchical queries instead of a flat ObjectStore namespace)
+<ul>
+<li>GetFileStatus, CreateDirectory, CreateFile, LookupFile</li>
+</ul>
+</li>
+<li>ACL related
+<ul>
+<li>Managing ACLs if <a href="../security/securityacls.html">internal ACLs</a> are used instead of <a href="../security/securitywithranger.html">Ranger</a></li>
+</ul>
+</li>
+<li>Delegation token (Get / Renew / Cancel)</li>
+<li>For security</li>
+<li>Admin APIs
+<ul>
+<li>Get S3 secret</li>
+<li>ServiceList (used for service discovery)</li>
+<li>DBUpdates (used by <a href="../feature/recon.html">Recon</a> downloads snapshots)</li>
+</ul>
+</li>
+</ul>
+<h3 id="persisted-state">Persisted state</h3>
+<p>The following data is persisted in Ozone Manager side in a specific RocksDB directory:</p>
+<ul>
+<li>Volume / Bucket / Key tables
+<ul>
+<li>This is the main responsibility of OM</li>
+<li>Key metadata contains the block id (which includes container id) to find the data</li>
+</ul>
+</li>
+<li>OpenKey table
+<ul>
+<li>for keys which are created, but not yet committed</li>
+</ul>
+</li>
+<li>Delegation token table
+<ul>
+<li>for security</li>
+</ul>
+</li>
+<li>PrefixInfo table
+<ul>
+<li>specific index table to store directory level ACL and to provide better performance for hierarchical queries</li>
+</ul>
+</li>
+<li>S3 secret table
+<ul>
+<li>For S3 secret management</li>
+</ul>
+</li>
+<li>Multipart info table
+<ul>
+<li>Inflight uploads should be tracked</li>
+</ul>
+</li>
+<li>Deleted table</li>
+<li>To track the blocks which should be deleted from the datanodes</li>
+</ul>
+<h2 id="notable-configurations">Notable configurations</h2>
+<table>
+<thead>
+<tr>
+<th>key</th>
+<th>default</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>ozone.om.address</td>
+<td>0.0.0.0:9862</td>
+<td>RPC address of the OM. Required by the client.</td>
+</tr>
+<tr>
+<td>ozone.om.http-address</td>
+<td>0.0.0.0:9874</td>
+<td>Default port of the HTTP server.</td>
+</tr>
+<tr>
+<td>ozone.metadata.dirs</td>
+<td>none</td>
+<td>Directory to store persisted data (RocksDB).</td>
+</tr>
+</tbody>
+</table>
+
+
+
+ <a class="btn btn-success btn-lg" href="../concept/storagecontainermanager.html">Next >></a>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="push"></div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: August 20, 2021 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/52ec27cf19e1d360c1d64bf7fc6181173124192a">52ec27cf1</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+</body>
+
+</html>
\ No newline at end of file
diff --git a/docs/1.2.0/concept/recon.html b/docs/1.2.0/concept/recon.html
new file mode 100644
index 0000000..deb7779
--- /dev/null
+++ b/docs/1.2.0/concept/recon.html
@@ -0,0 +1,619 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+ <div class="wrapper">
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="active">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
+
+
+
+ <div class="col-md-9">
+ <nav aria-label="breadcrumb">
+ <ol class="breadcrumb">
+ <li class="breadcrumb-item"><a href="../index.html">Home</a></li>
+ <li class="breadcrumb-item" aria-current="page"><a href="../concept.html">Architecture</a></li>
+ <li class="breadcrumb-item active" aria-current="page">Recon</li>
+ </ol>
+ </nav>
+
+
+
+<div class="pull-right">
+
+
+
+
+
+ <a href="../zh/concept/recon.html"><span class="label label-success">中文</span></a>
+
+
+</div>
+
+
+ <div class="col-md-9">
+ <h1>Recon</h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<p>Recon serves as a management and monitoring console for Ozone. It gives a
+bird’s-eye view of Ozone and helps users troubleshoot any issues by presenting
+the current state of the cluster through REST based APIs and rich web UI.</p>
+<h2 id="high-level-design">High Level Design</h2>
+<figure><img src="../concept/ReconHighLevelDesign.png" width="800px"/>
+</figure>
+
+<br/>
+<p>On a high level, Recon collects and aggregates metadata from Ozone Manager (OM),
+Storage Container Manager (SCM) and Datanodes (DN) and acts as a central
+management and monitoring console. Ozone administrators can use Recon to query
+the current state of the system without overloading OM or SCM.</p>
+<p>Recon maintains multiple databases to enable batch processing, faster querying
+and to persist aggregate information. It maintains a local copy of OM db and
+SCM db along with a SQL database for persisting aggregate information.</p>
+<p>Recon also integrates with Prometheus to provide a HTTP endpoint to query Prometheus
+for Ozone metrics and also to display a few crucial point in time metrics in
+the web UI.</p>
+<h2 id="recon-and-ozone-manager">Recon and Ozone Manager</h2>
+<figure><img src="../concept/ReconOmDesign.png" width="800px"/>
+</figure>
+
+<br/>
+<p>Recon gets a full snapshot of OM rocks db initially from the leader OM’s HTTP
+endpoint, untars the file and initializes RocksDB for querying locally. The
+database is kept in sync by periodically requesting delta updates from the leader
+OM via RPC calls from the last applied sequence id. If for any reason, the delta
+updates could not be retrieved or applied to the local db, a full snapshot is
+requested again to keep the local db in sync with OM db. Due to this, Recon can
+show stale information since the local db will not always be in sync.</p>
+<p>The db updates retrieved from OM is then converted into a batch of events for
+further processing by OM db tasks via <a href="#task-framework">Recon Task Framework</a>.</p>
+<h2 id="recon-and-storage-container-manager">Recon and Storage Container Manager</h2>
+<figure><img src="../concept/ReconScmDesign.png" width="800px"/>
+</figure>
+
+<br/>
+<p>Recon also acts as a passive SCM for datanodes. When Recon is configured in the
+cluster, all the datanodes register with Recon and send heartbeats, container
+reports, incremental container reports etc. to Recon similar to SCM. Recon uses
+all the information it gets from datanodes to construct its own copy of SCM rocks db
+locally. Recon never sends any command to datanodes in response and just acts as
+a passive SCM for faster lookup of SCM metadata.</p>
+<h2 id="a-nametask-frameworka-task-framework"><a name="task-framework"></a> Task Framework</h2>
+<p>Recon has its own Task framework to enable batch processing of data obtained
+from OM and SCM. A task can listen to and act upon db events such as <code>PUT</code>, <code>DELETE</code>,
+<code>UPDATE</code>, etc. on either OM db or SCM db. Based on this, a task either implements
+<code>org.apache.hadoop.ozone.recon.tasks.ReconOmTask</code> or extends
+<code>org.apache.hadoop.ozone.recon.scm.ReconScmTask</code>.</p>
+<p>An example <code>ReconOmTask</code> is <code>ContainerKeyMapperTask</code> that persists the container -> key
+mapping in RocksDB. This is useful to understand which keys were part of the container
+when the container is reported missing or is in a bad health state. Another example is
+<code>FileSizeCountTask</code> which keeps track of count of files within a given file size range in
+a SQL database. These tasks have implementations for two scenarios:</p>
+<ul>
+<li>Full snapshot (reprocess())</li>
+<li>Delta updates (process())</li>
+</ul>
+<p>When a full snapshot of OM db is obtained from the leader OM, the reprocess()
+is called on all the registered OM tasks. On subsequent delta updates, process()
+is called on these OM tasks.</p>
+<p>An example <code>ReconScmTask</code> is <code>ContainerHealthTask</code> that runs in configurable
+intervals to scan the list of all the containers and to persist the state of
+unhealthy containers (<code>MISSING</code>, <code>MIS_REPLICATED</code>, <code>UNDER_REPLICATED</code>, <code>OVER_REPLICATED</code>)
+in a SQL table. This information is used to determine if there are any missing
+containers in the cluster.</p>
+<h2 id="recon-and-prometheus">Recon and Prometheus</h2>
+<p>Recon can integrate with any Prometheus instance configured to collected metrics
+and can display useful information in Recon UI in Datanodes and Pipelines pages.
+Recon also exposes a proxy endpoint (<a href="../interface/reconapi.html#metrics">/metrics</a>)
+to query Prometheus. This integration can be enabled by setting this configuration <code>ozone.recon.prometheus.http.endpoint</code>
+to the Prometheus endpoint like <code>ozone.recon.prometheus.http.endpoint=localhost:9090</code>.</p>
+<h2 id="api-reference">API Reference</h2>
+<p><a href="../interface/reconapi.html">Link to complete API Reference</a></p>
+<h2 id="persisted-state">Persisted state</h2>
+<ul>
+<li>
+<p>A local copy of <a href="../concept/ozonemanager.html#persisted-state">OM database</a></p>
+</li>
+<li>
+<p>A local copy of <a href="../concept/storagecontainermanager.html#persisted-state">SCM database</a></p>
+</li>
+<li>
+<p>The following data is persisted in Recon in the specified RocksDB directory:</p>
+<ul>
+<li>ContainerKey table
+<ul>
+<li>Stores the mapping (container, key) -> count</li>
+</ul>
+</li>
+<li>ContainerKeyCount table
+<ul>
+<li>Stores containerID -> no. of keys count within the container</li>
+</ul>
+</li>
+</ul>
+</li>
+<li>
+<p>The following data is stored in the configured SQL database (default is Derby):</p>
+<ul>
+<li>GlobalStats table
+<ul>
+<li>A Key -> Value table to store aggregate information like total
+number of volumes / buckets / keys present in the cluster</li>
+</ul>
+</li>
+<li>FileCountBySize table
+<ul>
+<li>Keeps track of the number of files present within a file size range in the cluster</li>
+</ul>
+</li>
+<li>ReconTaskStatus table
+<ul>
+<li>Keeps track of the status and last run timestamp of the registered OM and SCM
+db tasks in the <a href="#task-framework">Recon Task Framework</a></li>
+</ul>
+</li>
+<li>ContainerHistory table
+<ul>
+<li>Stores ContainerReplica -> Datanode mapping with last known timestamp. This
+is used to determine the last known datanodes when a container is reported missing</li>
+</ul>
+</li>
+<li>UnhealthyContainers table
+<ul>
+<li>Keeps track of all the Unhealthy Containers (MISSING, UNDER_REPLICATED,
+OVER_REPLICATED, MIS_REPLICATED) in the cluster at any given time</li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+<h2 id="notable-configurations">Notable configurations</h2>
+<table>
+<thead>
+<tr>
+<th>key</th>
+<th>default</th>
+<th><div style="width: 300px;">description</div></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>ozone.recon.http-address</td>
+<td>0.0.0.0:9888</td>
+<td>The address and the base port where the Recon web UI will listen on.</td>
+</tr>
+<tr>
+<td>ozone.recon.address</td>
+<td>0.0.0.0:9891</td>
+<td>RPC address of the Recon.</td>
+</tr>
+<tr>
+<td>ozone.recon.db.dir</td>
+<td>none</td>
+<td>Directory where the Recon Server stores its metadata.</td>
+</tr>
+<tr>
+<td>ozone.recon.om.db.dir</td>
+<td>none</td>
+<td>Directory where the Recon Server stores its OM snapshot DB.</td>
+</tr>
+<tr>
+<td>ozone.recon.om.snapshot<br>.task.interval.delay</td>
+<td>10m</td>
+<td>Interval in MINUTES by Recon to request OM DB Snapshot / delta updates.</td>
+</tr>
+<tr>
+<td>ozone.recon.task<br>.missingcontainer.interval</td>
+<td>300s</td>
+<td>Time interval of the periodic check for Unhealthy Containers in the cluster.</td>
+</tr>
+<tr>
+<td>ozone.recon.sql.db.jooq.dialect</td>
+<td>DERBY</td>
+<td>Please refer to <a href="https://www.jooq.org/javadoc/latest/org.jooq/org/jooq/SQLDialect.html">SQL Dialect</a> to specify a different dialect.</td>
+</tr>
+<tr>
+<td>ozone.recon.sql.db.jdbc.url</td>
+<td>jdbc:derby:${ozone.recon.db.dir}<br>/ozone_recon_derby.db</td>
+<td>Recon SQL database jdbc url.</td>
+</tr>
+<tr>
+<td>ozone.recon.sql.db.username</td>
+<td>none</td>
+<td>Recon SQL database username.</td>
+</tr>
+<tr>
+<td>ozone.recon.sql.db.password</td>
+<td>none</td>
+<td>Recon SQL database password.</td>
+</tr>
+<tr>
+<td>ozone.recon.sql.db.driver</td>
+<td>org.apache.derby.jdbc<br>.EmbeddedDriver</td>
+<td>Recon SQL database jdbc driver.</td>
+</tr>
+</tbody>
+</table>
+
+
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="push"></div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: November 24, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/1a304ba81c9d52e2fb3a67e669d08c729a4113ae">1a304ba81</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+</body>
+
+</html>
\ No newline at end of file
diff --git a/docs/1.2.0/concept/storagecontainermanager.html b/docs/1.2.0/concept/storagecontainermanager.html
new file mode 100644
index 0000000..4f4703e
--- /dev/null
+++ b/docs/1.2.0/concept/storagecontainermanager.html
@@ -0,0 +1,532 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+ <div class="wrapper">
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="active">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-10 col-sm-offset-2 col-md-10 col-md-offset-2 main">
+
+
+
+ <div class="col-md-9">
+ <nav aria-label="breadcrumb">
+ <ol class="breadcrumb">
+ <li class="breadcrumb-item"><a href="../index.html">Home</a></li>
+ <li class="breadcrumb-item" aria-current="page"><a href="../concept.html">Architecture</a></li>
+ <li class="breadcrumb-item active" aria-current="page">Storage Container Manager</li>
+ </ol>
+ </nav>
+
+
+
+<div class="pull-right">
+
+
+
+
+
+ <a href="../zh/concept/storagecontainermanager.html"><span class="label label-success">中文</span></a>
+
+
+</div>
+
+
+ <div class="col-md-9">
+ <h1>Storage Container Manager</h1>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<p>Storage Container Manager (SCM) is the leader node of the <em>block space management</em>. The main responsibility is to create and manage <a href="../concept/containers.html">containers</a> which is the main replication unit of Ozone.</p>
+<p><img src="StorageContainerManager.png" alt="Storage Container Manager"></p>
+<h2 id="main-responsibilities">Main responsibilities</h2>
+<p>Storage container manager provides multiple critical functions for the Ozone
+cluster. SCM acts as the cluster manager, Certificate authority, Block
+manager and the Replica manager.</p>
+<p>SCM is in charge of creating an Ozone cluster. When an SCM is booted up via <code>init</code> command, SCM creates the cluster identity and root certificates needed for the SCM certificate authority. SCM manages the life cycle of a data node in the cluster.</p>
+<ol>
+<li>
+<p>SCM is the block manager. SCM
+allocates blocks and assigns them to data nodes. Clients
+read and write these blocks directly.</p>
+</li>
+<li>
+<p>SCM keeps track of all the block
+replicas. If there is a loss of data node or a disk, SCM
+detects it and instructs data nodes make copies of the
+missing blocks to ensure high availability.</p>
+</li>
+<li>
+<p><strong>SCM’s Certificate Authority</strong> is in
+charge of issuing identity certificates for each and every
+service in the cluster. This certificate infrastructure makes
+it easy to enable mTLS at network layer and the block
+token infrastructure depends on this certificate infrastructure.</p>
+</li>
+</ol>
+<h2 id="main-components">Main components</h2>
+<p>For a detailed view of Storage Container Manager this section gives a quick overview about the provided network services and the stored persisted data.</p>
+<h3 id="network-services-provided-by-storage-container-manager">Network services provided by Storage Container Manager:</h3>
+<ul>
+<li>Pipelines: List/Delete/Activate/Deactivate
+<ul>
+<li>pipelines are set of datanodes to form replication groups</li>
+<li>Raft groups are planned by SCM</li>
+</ul>
+</li>
+<li>Containers: Create / List / Delete containers</li>
+<li>Admin related requests</li>
+<li>Safemode status/modification</li>
+<li>Replication manager start / stop</li>
+<li>CA authority service</li>
+<li>Required by other sever components</li>
+<li>Datanode HeartBeat protocol
+<ul>
+<li>From Datanode to SCM (30 sec by default)</li>
+<li>Datanodes report the status of containers, node…</li>
+<li>SCM can add commands to the response</li>
+</ul>
+</li>
+</ul>
+<p>Note: client doesn’t connect directly to the SCM</p>
+<h3 id="persisted-state">Persisted state</h3>
+<p>The following data is persisted in Storage Container Manager side in a specific RocksDB directory</p>
+<ul>
+<li>Pipelines
+<ul>
+<li>Replication group of servers. Maintained to find a group for new container/block allocations.</li>
+</ul>
+</li>
+<li>Containers
+<ul>
+<li>Containers are the replication units. Data is required to act in case of data under/over replicated.</li>
+</ul>
+</li>
+<li>Deleted blocks
+<ul>
+<li>Block data is deleted in the background. Need a list to follow the progress.</li>
+</ul>
+</li>
+<li>Valid cert, Revoked certs</li>
+<li>Used by the internal Certificate Authority to authorize other Ozone services</li>
+</ul>
+<h2 id="notable-configurations">Notable configurations</h2>
+<table>
+<thead>
+<tr>
+<th>key</th>
+<th>default</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>ozone.scm.container.size</td>
+<td>5GB</td>
+<td>Default container size used by Ozone</td>
+</tr>
+<tr>
+<td>ozone.scm.block.size</td>
+<td>256MB</td>
+<td>The default size of a data block.</td>
+</tr>
+<tr>
+<td>hdds.scm.safemode.min.datanode</td>
+<td>1</td>
+<td>Minimum number of datanodes to start the real work.</td>
+</tr>
+<tr>
+<td>ozone.scm.http-address</td>
+<td>0.0.0.0:9876</td>
+<td>HTTP address of the SCM server</td>
+</tr>
+<tr>
+<td>ozone.metadata.dirs</td>
+<td>none</td>
+<td>Directory to store persisted data (RocksDB).</td>
+</tr>
+</tbody>
+</table>
+
+
+
+ <a class="btn btn-success btn-lg" href="../concept/containers.html">Next >></a>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+ </div>
+ <div class="push"></div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: November 24, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/1a304ba81c9d52e2fb3a67e669d08c729a4113ae">1a304ba81</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+</body>
+
+</html>
\ No newline at end of file
diff --git a/docs/1.2.0/css/bootstrap-theme.min.css b/docs/1.2.0/css/bootstrap-theme.min.css
new file mode 100644
index 0000000..2a69f48
--- /dev/null
+++ b/docs/1.2.0/css/bootstrap-theme.min.css
@@ -0,0 +1,6 @@
+/*!
+ * Bootstrap v3.4.1 (https://getbootstrap.com/)
+ * Copyright 2011-2019 Twitter, Inc.
+ * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
+ */.btn-danger,.btn-default,.btn-info,.btn-primary,.btn-success,.btn-warning{text-shadow:0 -1px 0 rgba(0,0,0,.2);-webkit-box-shadow:inset 0 1px 0 rgba(255,255,255,.15),0 1px 1px rgba(0,0,0,.075);box-shadow:inset 0 1px 0 rgba(255,255,255,.15),0 1px 1px rgba(0,0,0,.075)}.btn-danger.active,.btn-danger:active,.btn-default.active,.btn-default:active,.btn-info.active,.btn-info:active,.btn-primary.active,.btn-primary:active,.btn-success.active,.btn-success:active,.btn-warning.active,.btn-warnin [...]
+/*# sourceMappingURL=bootstrap-theme.min.css.map */
\ No newline at end of file
diff --git a/docs/1.2.0/css/bootstrap-theme.min.css.map b/docs/1.2.0/css/bootstrap-theme.min.css.map
new file mode 100644
index 0000000..5d75106
--- /dev/null
+++ b/docs/1.2.0/css/bootstrap-theme.min.css.map
@@ -0,0 +1 @@
+{"version":3,"sources":["bootstrap-theme.css","dist/css/bootstrap-theme.css","less/theme.less","less/mixins/vendor-prefixes.less","less/mixins/gradients.less","less/mixins/reset-filter.less"],"names":[],"mappings":"AAAA;;;;ACUA,YCWA,aDbA,UAFA,aACA,aAEA,aCkBE,YAAA,EAAA,KAAA,EAAA,eC2CA,mBAAA,MAAA,EAAA,IAAA,EAAA,qBAAA,CAAA,EAAA,IAAA,IAAA,iBACQ,WAAA,MAAA,EAAA,IAAA,EAAA,qBAAA,CAAA,EAAA,IAAA,IAAA,iBF7CV,mBANA,mBACA,oBCWE,oBDRF,iBANA,iBAIA,oBANA,oBAOA,oBANA,oBAQA,oBANA,oBEmDE,mBAAA,MAAA,EAAA,IA [...]
\ No newline at end of file
diff --git a/docs/1.2.0/css/bootstrap.min.css b/docs/1.2.0/css/bootstrap.min.css
new file mode 100644
index 0000000..5b96335
--- /dev/null
+++ b/docs/1.2.0/css/bootstrap.min.css
@@ -0,0 +1,6 @@
+/*!
+ * Bootstrap v3.4.1 (https://getbootstrap.com/)
+ * Copyright 2011-2019 Twitter, Inc.
+ * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
+ *//*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */html{font-family:sans-serif;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,details,figcaption,figure,footer,header,hgroup,main,menu,nav,section,summary{display:block}audio,canvas,progress,video{display:inline-block;vertical-align:baseline}audio:not([controls]){display:none;height:0}[hidden],template{display:none}a{background-color:transparent}a:active,a:hover{outline:0}abbr [...]
+/*# sourceMappingURL=bootstrap.min.css.map */
\ No newline at end of file
diff --git a/docs/1.2.0/css/bootstrap.min.css.map b/docs/1.2.0/css/bootstrap.min.css.map
new file mode 100644
index 0000000..0ae3de5
--- /dev/null
+++ b/docs/1.2.0/css/bootstrap.min.css.map
@@ -0,0 +1 @@
+{"version":3,"sources":["bootstrap.css","less/normalize.less","dist/css/bootstrap.css","less/print.less","less/glyphicons.less","less/scaffolding.less","less/mixins/vendor-prefixes.less","less/mixins/tab-focus.less","less/mixins/image.less","less/type.less","less/mixins/text-emphasis.less","less/mixins/background-variant.less","less/mixins/text-overflow.less","less/code.less","less/grid.less","less/mixins/grid.less","less/mixins/grid-framework.less","less/tables.less","less/mixins/table- [...]
\ No newline at end of file
diff --git a/docs/1.2.0/css/ozonedoc.css b/docs/1.2.0/css/ozonedoc.css
new file mode 100644
index 0000000..90068cc
--- /dev/null
+++ b/docs/1.2.0/css/ozonedoc.css
@@ -0,0 +1,212 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Base structure
+ */
+
+html, body {
+ height: 100%;
+ margin: 0;
+}
+
+/* Move down content because we have a fixed navbar that is 50px tall */
+body {
+ padding-top: 50px;
+ font-size: 14pt;
+}
+
+pre {
+ font-size: 14pt;
+}
+
+a {
+ color: #448628;
+}
+
+a:hover {
+ color: #245F0B;
+}
+/*
+ * Global add-ons
+ */
+
+.sub-header {
+ padding-bottom: 10px;
+ border-bottom: 1px solid #eee;
+}
+
+/*
+ * Top navigation
+ * Hide default border to remove 1px line.
+ */
+.navbar-fixed-top {
+ border: 0;
+}
+
+A.ozone-logo {
+ height: 50px;
+ padding: 5px 10px 5px 0px;
+}
+
+.ozone-logo img {
+ width: 40px;
+ padding: 0px;
+}
+
+/*
+ * Sidebar
+ */
+
+/* Hide for mobile, show later */
+.sidebar {
+ display: none;
+}
+
+@media (min-width: 768px) {
+ .sidebar {
+ position: fixed;
+ top: 51px;
+ bottom: 0;
+ left: 0;
+ z-index: 1000;
+ display: block;
+ padding: 20px;
+ overflow-x: hidden;
+ overflow-y: auto; /* Scrollable contents if viewport is shorter than content. */
+ background-color: #f5f5f5;
+ border-right: 1px solid #eee;
+ }
+}
+
+/* Sidebar navigation */
+.nav-sidebar {
+ margin-right: -21px; /* 20px padding + 1px border */
+ margin-bottom: 20px;
+ margin-left: -20px;
+}
+.nav-sidebar > li > a {
+ padding-right: 20px;
+ padding-left: 20px;
+}
+.nav-sidebar > li > ul > li > a {
+ padding-right: 40px;
+ padding-left: 40px;
+}
+.nav-sidebar > li > ul > li > ul > li > a {
+ padding-right: 60px;
+ padding-left: 60px;
+}
+.nav-sidebar .active > a,
+.nav-sidebar .active > a:hover,
+.nav-sidebar .active > a:focus {
+ color: #fff;
+ background-color: #73B148;
+}
+
+/*
+ * Main content
+ */
+
+.main {
+ padding: 20px;
+}
+@media (min-width: 768px) {
+ .main {
+ padding-right: 40px;
+ padding-left: 40px;
+ }
+}
+.main .page-header {
+ margin-top: 0;
+}
+
+
+/*
+ * Placeholder dashboard ideas
+ */
+
+.placeholders {
+ margin-bottom: 30px;
+ text-align: center;
+}
+.placeholders h4 {
+ margin-bottom: 0;
+}
+.placeholder {
+ margin-bottom: 20px;
+}
+.placeholder img {
+ display: inline-block;
+ border-radius: 50%;
+}
+
+h4 {
+ font-weight: bold;
+}
+
+.cardgroup {
+ margin-bottom: 50px;
+}
+
+.cardgroup .card {
+ padding: 20px;
+}
+
+.cardgroup .media {
+ padding: 30px;
+}
+
+h1 {
+ margin-bottom: 20px;
+}
+
+.card {
+ padding: 20px;
+}
+
+figure.ozone-usage {
+ padding: 40px;
+}
+
+table.table {
+ margin: 20px 20px 40px;
+}
+
+.footer,
+.push {
+ height: 50px;
+}
+
+.footer {
+ background-color: #f5f5f5;
+}
+
+.wrapper {
+ min-height: 100%;
+
+ /* Equal to height of footer */
+ /* But also accounting for potential margin-bottom of last child */
+ margin-bottom: -50px;
+}
+
+.footer .container {
+ padding-top: 10px;
+ padding-bottom: 10px;
+ text-align: center;
+}
diff --git a/docs/1.2.0/design.html b/docs/1.2.0/design.html
new file mode 100644
index 0000000..532f5ae
--- /dev/null
+++ b/docs/1.2.0/design.html
@@ -0,0 +1,649 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="./css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="./css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+<body>
+
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="./index.html" class="navbar-left ozone-logo">
+ <img src="./ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="./index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+
+<div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="./index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="./start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="./concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="./feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="./interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="./security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="./security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="./security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="./tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="./recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="./design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+
+ <main>
+ <main>
+ <h1>Available design docs</h1>
+
+ <table class="table">
+ <thead>
+ <tr>
+ <td>Date</td>
+ <td>JIRA</td>
+ <td>Title</td>
+ <td>Summary</td>
+ <td>Status</td>
+ </tr>
+ </thead>
+ <tbody>
+
+ <tr>
+ <td>2021-04-12</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-2939">HDDS-2939</a>
+ </td>
+ <td><a href="./design/namespace-support.html">Ozone FS namespace / prefix table</a></td>
+ <td>Use additional prefix table for indexed data retrieval</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2021-02-15</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-3698">HDDS-3698</a>
+ </td>
+ <td><a href="./design/nonrolling-upgrade.html">Non Rolling upgrade support in Ozone.</a></td>
+ <td>‘Non rolling’ upgrade support in Ozone.</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2021-02-15</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-4470">HDDS-4470</a>
+ </td>
+ <td><a href="./design/omprepare.html">Ozone Manager Prepare for Upgrade</a></td>
+ <td>‘Prepare upgrade’ step to make sure all the OMs use the same version of the software to update their DBs (apply transaction) for a given request.</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2021-02-15</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/"></a>
+ </td>
+ <td><a href="./design/upgrade-dev-primer.html">Upgrade - Developer Primer</a></td>
+ <td>Helpful resources for those who are bringing layout changes.</td>
+ <td></td>
+ </tr>
+
+ <tr>
+ <td>2020-11-09</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-4440">HDDS-4440</a>
+ </td>
+ <td><a href="./design/s3-performance.html">Proposed persistent OM connection for S3 gateway</a></td>
+ <td>Proposal to use per-request authentication and persistent connections between S3g and OM</td>
+ <td>accepted</td>
+ </tr>
+
+ <tr>
+ <td>2020-06-30</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-3816">HDDS-3816</a>
+ </td>
+ <td><a href="./design/ec.html">Erasure Coding in Ozone</a></td>
+ <td>Use Erasure Coding algorithm for efficient storage</td>
+ <td>draft</td>
+ </tr>
+
+ <tr>
+ <td>2020-06-30</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-2665">HDDS-2665</a>
+ </td>
+ <td><a href="./design/ofs.html">Implement new Ozone FileSystem scheme ofs://</a></td>
+ <td>A new schema structure for Hadoop compatible file system</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2020-06-08</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-3755">HDDS-3755</a>
+ </td>
+ <td><a href="./design/storage-class.html">Storage Class</a></td>
+ <td>New abstraction to configure replication methods.</td>
+ <td>draft</td>
+ </tr>
+
+ <tr>
+ <td>2020-04-02</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-3331">HDDS-3331</a>
+ </td>
+ <td><a href="./design/volume-management.html">Ozone Volume Management</a></td>
+ <td>A simplified version of mapping between S3 buckets and Ozone volume/buckets</td>
+ <td>accepted</td>
+ </tr>
+
+ <tr>
+ <td>2020-03-25</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-3001">HDDS-3001</a>
+ </td>
+ <td><a href="./design/nfs.html">NFS support Ozone</a></td>
+ <td>Adopt NFS gateway of HDFS and provide NFS file system view</td>
+ <td>draft</td>
+ </tr>
+
+ <tr>
+ <td>2020-03-05</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-2823">HDDS-2823</a>
+ </td>
+ <td><a href="./design/scmha.html">SCM HA support</a></td>
+ <td>HA for Storage Container Manager using Ratis to replicate data</td>
+ <td>implementing</td>
+ </tr>
+
+ <tr>
+ <td>2020-01-09</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-2867">HDDS-2867</a>
+ </td>
+ <td><a href="./design/token.html">Generic Extensible Token support for Ozone</a></td>
+ <td>Extend existing token architecture to all entities of Ozone</td>
+ <td>accepted</td>
+ </tr>
+
+ <tr>
+ <td>2019-11-07</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-2416">HDDS-2416</a>
+ </td>
+ <td><a href="./design/trash.html">Ozone Trash Feature</a></td>
+ <td>Feature provides a user with the ability to recover keys that may have been deleted accidentally. (similar to the HDFS trash feature).</td>
+ <td>implementing</td>
+ </tr>
+
+ <tr>
+ <td>2019-08-19</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1996">HDDS-1996</a>
+ </td>
+ <td><a href="./design/recon2.html">Recon server v2</a></td>
+ <td>Second phase of Recon development. Recon acts as a passive SCM.</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2019-07-31</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1881">HDDS-1881</a>
+ </td>
+ <td><a href="./design/decommissioning.html">Decommissioning in Ozone</a></td>
+ <td>Formal process to shut down machines in a safe way after the required replications.</td>
+ <td>implementing</td>
+ </tr>
+
+ <tr>
+ <td>2019-06-20</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-2012">HDDS-2012</a>
+ </td>
+ <td><a href="./design/gdpr.html">Ozone GDPR framework</a></td>
+ <td>Crypto key management to handle GDPR “right to be forgotten” feature</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2019-06-20</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1672">HDDS-1672</a>
+ </td>
+ <td><a href="./design/locks.html">Ozone locks in OM</a></td>
+ <td>Hierarchical locking structure for OM elements (volumes/keys/buckets)</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2019-06-07</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1659">HDDS-1659</a>
+ </td>
+ <td><a href="./design/ozone-enhancement-proposals.html">Ozone Enhancement Proposals</a></td>
+ <td>Definition of the process to share new technical proposals with the Ozone community.</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2019-05-25</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1467">HDDS-1467</a>
+ </td>
+ <td><a href="./design/configless.html">Configless Ozone service management</a></td>
+ <td>Distribute only minimal configuration and download all the remaining before start</td>
+ <td>accepted</td>
+ </tr>
+
+ <tr>
+ <td>2019-05-21</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1564">HDDS-1564</a>
+ </td>
+ <td><a href="./design/multiraft.html">Ozone multi-raft support</a></td>
+ <td>Datanodes can be part of multiple independent RAFT groups / pipelines</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2019-04-25</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1466">HDDS-1466</a>
+ </td>
+ <td><a href="./design/typesafeconfig.html">Type-safe configuration API</a></td>
+ <td>Inject configuration values based on annotations instead of using constants and Hadoop API</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2019-02-19</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1084">HDDS-1084</a>
+ </td>
+ <td><a href="./design/recon1.html">Recon server 1</a></td>
+ <td>Second phase of Recon development. Recon acts as a passive SCM.</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2019-02-08</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-1041">HDDS-1041</a>
+ </td>
+ <td><a href="./design/tde.html">Ozone Encryption At Rest</a></td>
+ <td>Transparent Data Encryption encrypts the data before store it</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2018-11-16</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-698">HDDS-698</a>
+ </td>
+ <td><a href="./design/topology.html">Topology-awareness</a></td>
+ <td>Placement policy to use rack information for read and write</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2018-09-27</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-434">HDDS-434</a>
+ </td>
+ <td><a href="./design/s3gateway.html">S3 protocol support for Ozone</a></td>
+ <td>Support any AWS S3 compatible client with dedicated REST endpoint</td>
+ <td>implemented</td>
+ </tr>
+
+ <tr>
+ <td>2018-09-18</td>
+ <td>
+ <a href="https://issues.apache.org/jira/browse/HDDS-505">HDDS-505</a>
+ </td>
+ <td><a href="./design/omha.html">Ozone Manager HA</a></td>
+ <td>Support HA for Ozone Manager with the help of RATIS</td>
+ <td>implemented</td>
+ </tr>
+
+ </tbody>
+ </table>
+ </main>
+ </main>
+
+ </div>
+ </div>
+</div>
+</div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="./js/jquery-3.5.1.min.js"></script>
+<script src="./js/ozonedoc.js"></script>
+<script src="./js/bootstrap.min.js"></script>
+
+
+</body>
+</html>
+
diff --git a/docs/1.2.0/design/configless.html b/docs/1.2.0/design/configless.html
new file mode 100644
index 0000000..d0a418c
--- /dev/null
+++ b/docs/1.2.0/design/configless.html
@@ -0,0 +1,398 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-1467">[HDDS-1467]</a> Configless Ozone service management (accepted) </h1>
+ <div><i>Authors: Márton Elek, Anu Engineer</i><div class="pull-right">2019-05-25</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Distribute only minimal configuration and download all the remaining before start
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>Configuration keys are partitioned to two parts: runtime settings and environment settings.</p>
+<p>Environment settings (hosts, ports) can be downloaded at start.</p>
+<h1 id="link">Link</h1>
+<ul>
+<li><a href="https://issues.apache.org/jira/secure/attachment/12966992/configless.pdf">https://issues.apache.org/jira/secure/attachment/12966992/configless.pdf</a></li>
+</ul>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: April 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/9872253e89377624dfea5fb9b321538fc70f91c4">9872253e8</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/decommissioning.html b/docs/1.2.0/design/decommissioning.html
new file mode 100644
index 0000000..031dca4
--- /dev/null
+++ b/docs/1.2.0/design/decommissioning.html
@@ -0,0 +1,1230 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-1881">[HDDS-1881]</a> Decommissioning in Ozone (implementing) </h1>
+ <div><i>Authors: Anu Engineer, Marton Elek, Stephen O'Donnell</i><div class="pull-right">2019-07-31</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Formal process to shut down machines in a safe way after the required replications.
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.</p>
+<p>There are two main classes of the decommissioning:</p>
+<ul>
+<li>
+<p><strong>Maintenance mode</strong>: where the node is expected to be back after a while. It may not require replication of containers if enough replicas are available from other nodes (as we expect to have the current replicas after the restart.)</p>
+</li>
+<li>
+<p><strong>Decommissioning</strong>: where the node won’t be started again. All the data should be replicated according to the current replication rules.</p>
+</li>
+</ul>
+<p>Goals:</p>
+<ul>
+<li>Decommissioning can be canceled any time</li>
+<li>The progress of the decommissioning should be trackable</li>
+<li>The nodes under decommissioning / maintenance mode should not been used for new pipelines / containers</li>
+<li>The state of the datanodes should be persisted / replicated by the SCM (in HDFS the decommissioning info exclude/include lists are replicated manually by the admin). If datanode is marked for decommissioning this state be available after SCM and/or Datanode restarts.</li>
+<li>We need to support validations before decommissioning (but the violations can be ignored by the admin).</li>
+<li>The administrator should be notified when a node can be turned off.</li>
+<li>The maintenance mode can be time constrained: if the node marked for maintenance for 1 week and the node is not up after one week, the containers should be considered as lost (DEAD node) and should be replicated.</li>
+</ul>
+<h1 id="introduction">Introduction</h1>
+<p>Ozone is a highly available file system that relies on commodity hardware. In other words, Ozone is designed to handle failures of these nodes all the time.</p>
+<p>The Storage Container Manager(SCM) is designed to monitor the node health and replicate blocks and containers as needed.</p>
+<p>At times, Operators of the cluster can help the SCM by giving it hints. When removing a datanode, the operator can provide a hint. That is, a planned failure of the node is coming up, and SCM can make sure it reaches a safe state to handle this planned failure.</p>
+<p>Some times, this failure is transient; that is, the operator is taking down this node temporarily. In that case, we can live with lower replica counts by being optimistic.</p>
+<p>Both of these operations, <strong>Maintenance</strong>, and <strong>Decommissioning</strong> are similar from the Replication point of view. In both cases, and the user instructs us on how to handle an upcoming failure.</p>
+<p>Today, SCM (<em>Replication Manager</em> component inside SCM) understands only one form of failure handling. This paper extends Replica Manager failure modes to allow users to request which failure handling model to be adopted(Optimistic or Pessimistic).</p>
+<p>Based on physical realities, there are two responses to any perceived failure, to heal the system by taking corrective actions or ignore the failure since the actions in the future will heal the system automatically.</p>
+<h2 id="user-experiences-decommissioning-vs-maintenance-mode">User Experiences (Decommissioning vs Maintenance mode)</h2>
+<p>From the user’s point of view, there are two kinds of planned failures that the user would like to communicate to Ozone.</p>
+<p>The first kind is when a ‘real’ failure is going to happen in the future. This ‘real’ failure is the act of decommissioning. We denote this as “decommission” throughout this paper. The response that the user wants is SCM/Ozone to make replicas to deal with the planned failure.</p>
+<p>The second kind is when the failure is ‘transient.’ The user knows that this failure is temporary and cluster in most cases can safely ignore this issue. However, if the transient failures are going to cause a failure of availability; then the user would like the Ozone to take appropriate actions to address it. An example of this case, is if the user put 3 data nodes into maintenance mode and switched them off.</p>
+<p>The transient failure can violate the availability guarantees of Ozone; Since the user is telling us not to take corrective actions. Many times, the user does not understand the impact on availability while asking Ozone to ignore the failure.</p>
+<p>So this paper proposes the following definitions for Decommission and Maintenance of data nodes.</p>
+<p><strong>Decommission</strong> <em>of a data node is deemed to be complete when SCM/Ozone completes the replica of all containers on decommissioned data node to other data nodes.That is, the expected count matches the healthy count of containers in the cluster</em>.</p>
+<p><strong>Maintenance mode</strong> of a data node is complete if Ozone can guarantee at <em>least one copy of every container</em> is available in other healthy data nodes.</p>
+<h2 id="examples">Examples</h2>
+<p>Here are some illustrative examples:</p>
+<ol>
+<li>
+<p>Let us say we have a container, which has only one copy and resides on Machine A. If the user wants to put machine A into maintenance mode; Ozone will make a replica before entering the maintenance mode.</p>
+</li>
+<li>
+<p>Suppose a container has two copies, and the user wants to put Machine A to maintenance mode. In this case; the Ozone understands that availability of the container is not affected and hence can decide to forgo replication.</p>
+</li>
+<li>
+<p>Suppose a container has two copies, and the user wants to put Machine A into maintenance mode. However, the user wants to put the machine into maintenance mode for one month. As the period of maintenance mode increases, the probability of data loss increases; hence, Ozone might choose to make a replica of the container even if we are entering maintenance mode.</p>
+</li>
+<li>
+<p>The semantics of decommissioning means that as long as we can find copies of containers in other machines, we can technically get away with calling decommission complete. Hence this clarification node; in the ordinary course of action; each decommission will create a replication flow for each container we have; however, it is possible to complete a decommission of a data node, even if we get a failure of the data node being decommissioned. As long as we can find the other datanodes t [...]
+</li>
+<li>
+<p>Let us say we have a copy of a container replica on Machine A, B, and C. It is possible to decommission all three machines at the same time, as decommissioning is just a status indicator of the data node and until we finish the decommissioning process.</p>
+</li>
+</ol>
+<p>The user-visible features for both of these are very similar:</p>
+<p>Both Decommission and Maintenance mode can be canceled any time before the operation is marked as completed by SCM.</p>
+<p>Decommissioned nodes, if and when added back, shall be treated as new data nodes; if they have blocks or containers on them, they can be used to reconstruct data.</p>
+<h2 id="maintenance-mode-in-hdfs">Maintenance mode in HDFS</h2>
+<p>HDFS supports decommissioning and maintenance mode similar to Ozone. This is a quick description of the HDFS approach.</p>
+<p>The usage of HDFS maintenance mode:</p>
+<ul>
+<li>First, you set a minimum replica count on the cluster, which can be zero, but defaults to 1.</li>
+<li>Then you can set a number of nodes into maintenance, with an expiry time or have them remain in maintenance forever, until they are manually removed. Nodes are put into maintenance in much the same way as nodes are decommissioned.</li>
+<li>When a set of nodes go into maintenance, all blocks hosted on them are scanned and if the node going into maintenance would cause the number of replicas to fall below the minimum replica count, the relevant nodes go into a decommissioning like state while new replicas are made for the blocks.</li>
+<li>Once the node goes into maintenance, it can be stopped etc and HDFS will not be concerned about the under-replicated state of the blocks.</li>
+<li>When the expiry time passes, the node is put back to normal state (if it is online and heartbeating) or marked as dead, at which time new replicas will start to be made.</li>
+</ul>
+<p>This is very similar to decommissioning, and the code to track maintenance mode and ensure the blocks are replicated etc, is effectively the same code as with decommissioning. The one area that differs is probably in the replication monitor as it must understand that the node is expected to be offline.</p>
+<p>The ideal way to use maintenance mode, is when you know there are a set of nodes you can stop without having to do any replications. In HDFS, the rack awareness states that all blocks should be on two racks, so that means a rack can be put into maintenance safely.</p>
+<p>There is another feature in HDFS called “upgrade Domain” which allows each datanode to be assigned a group. By default there should be at least 3 groups (domains) and then each of the 3 replicas will be stored on different group, allowing one full group to be put into maintenance at once. That is not yet supported in CDH, but is something we are targeting for CDPD I believe.</p>
+<p>One other difference with maintenance mode and decommissioning, is that you must have some sort of monitor thread checking for when maintenance is scheduled to end. HDFS solves this by having a class called the DatanodeAdminManager, and it tracks all nodes transitioning state, the under-replicated block count on them etc.</p>
+<h1 id="implementation">Implementation</h1>
+<h2 id="datanode-state-machine">Datanode state machine</h2>
+<p><code>NodeStateManager</code> maintains the state of the connected datanodes. The possible states:</p>
+<table>
+<thead>
+<tr>
+<th>state</th>
+<th>description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>The node is up and running.</td>
+</tr>
+<tr>
+<td>STALE</td>
+<td>Some heartbeats were missing for an already missing nodes.</td>
+</tr>
+<tr>
+<td>DEAD</td>
+<td>The stale node has not been recovered.</td>
+</tr>
+<tr>
+<td>ENTERING_MAINTENANCE</td>
+<td>The in-progress state, scheduling is disabled but the node can’t not been turned off due to in-progress replication.</td>
+</tr>
+<tr>
+<td>IN_MAINTENANCE</td>
+<td>Node can be turned off but we expected to get it back and have all the replicas.</td>
+</tr>
+<tr>
+<td>DECOMMISSIONING</td>
+<td>The in-progress state, scheduling is disabled, all the containers should be replicated to other nodes.</td>
+</tr>
+<tr>
+<td>DECOMMISSIONED</td>
+<td>The node can be turned off, all the containers are replicated to other machine</td>
+</tr>
+</tbody>
+</table>
+<h2 id="high-level-algorithm">High level algorithm</h2>
+<p>The Algorithm is pretty simple from the Decommission or Maintenance point of view;</p>
+<ol>
+<li>
+<p>Mark a data node as DECOMMISSIONING or ENTERING_MAINTENANCE. This implies that node is NOT healthy anymore; we assume the use of a single flag and law of excluded middle.</p>
+</li>
+<li>
+<p>Pipelines should be shut down and wait for confirmation that all pipelines are shutdown. So no new I/O or container creation can happen on a Datanode that is part of decomm/maint.</p>
+</li>
+<li>
+<p>Once the Node has been marked as DECOMMISSIONING or ENTERING_MAINTENANCE; the Node will generate a list of containers that need replication. This list is generated by the Replica Count decisions for each container; the Replica Count will be computed by Replica Manager;</p>
+</li>
+<li>
+<p>Replica Manager will check the stop condition for each node. The following should be true for all the containers to go from DECOMMISSIONING to DECOMMISSIONED or from ENTERING_MAINTENANCE to IN_MAINTENANCE.</p>
+</li>
+</ol>
+<ul>
+<li>Container is closed.</li>
+<li>We have at least one HEALTHY copy at all times.</li>
+<li>For entering DECOMMISSIONED mode <code>maintenance + healthy</code> must equal to <code>expectedCount</code></li>
+</ul>
+<ol start="5">
+<li>We will update the node state to DECOMMISSIONED or IN_MAINTENANCE reached state.</li>
+</ol>
+<p><em>Replica count</em> is a calculated number which represents the number of <em>missing</em> replicas. The number can be negative in case of an over-replicated container.</p>
+<h2 id="calculation-of-the-_replica-count_-required-replicas">Calculation of the <em>Replica count</em> (required replicas)</h2>
+<h3 id="counters--variables">Counters / Variables</h3>
+<p>We have 7 different datanode state, but some of them can be combined. At high level we can group the existing state to three groups:</p>
+<ul>
+<li>healthy state (when the container is available)</li>
+<li>maintenance (including IN_MAINTENANCE and ENTERING_MAINTENANCE)</li>
+<li>all the others.</li>
+</ul>
+<p>To calculate the required steps (required replication + stop condition) we need counter about the first two categories.</p>
+<p>Each counters should be calculated per container bases.</p>
+<table>
+<thead>
+<tr>
+<th>Node state</th>
+<th>Variable (# of containers)</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td><code>healthy</code></td>
+</tr>
+<tr>
+<td>STALE + DEAD + DECOMMISSIONED</td>
+<td></td>
+</tr>
+<tr>
+<td>DECOMMISSIONING</td>
+<td></td>
+</tr>
+<tr>
+<td>ENTERING_MAINTENANCE + IN_MAINTENANCE</td>
+<td><code>maintenance</code></td>
+</tr>
+</tbody>
+</table>
+<h3 id="the-current-replication-model">The current replication model</h3>
+<p>The current replication model in SCM/Ozone is very simplistic. We compute the replication count or the number of replications that we need to do as:</p>
+<pre tabindex="0"><code>Replica count = expectedCount - currentCount
+</code></pre><p>In case the <em>Replica count</em> is positive, it means that we need to make more replicas. If the number is negative, it means that we are over replicated and we need to remove some replicas of this container. If the Replica count for a container is zero; it means that we have the expected number of containers in the cluster.</p>
+<p>To support idempotent placement strategies we should subtract the in-flight replications from the result: If there are one in-flight replication process and two replicas we won’t start a new replication command unless the original command is timed out. The timeout is configured with <code>hdds.scm.replication.event.timeout</code> and the default value is 10 minutes.</p>
+<p>More preciously the current algorithm is the following:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">replicaCount <span style="color:#f92672">=</span> expectedCount <span style="color:#f92672">-</span> healthy<span style="color:#f92672">;</span>
+
+<span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>replicaCount <span style="color:#f92672">-</span> inflight copies <span style="color:#f92672">+</span> inflight deletes<span style="color:#f92672">)</span> <span style="color:#f92672">></span> 0 <span style="color:#f92672">{</span>
+ <span style="color:#75715e">// container is over replicated
+</span><span style="color:#75715e"></span><span style="color:#f92672">}.</span><span style="color:#a6e22e">else</span> <span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>replicaCount <span style="color:#f92672">-</span> inflight copies <span style="color:#f92672">+</span> inflight deletes<span style="color:#f92672">)</span> <span style="color:#f92672"><</span>0 <span style="color:#f92672">{</span>
+ <span style="color:#75715e">// handle under replicated containers
+</span><span style="color:#75715e"></span><span style="color:#f92672">}</span>
+</code></pre></div><p>The handling of inflight copies and deletes are independent from the decommissioning problem therefore here we focus only on the core mode:</p>
+<pre tabindex="0"><code>replicaCount = expectedCount - healthy;
+</code></pre><h3 id="the-proposed-solution">The proposed solution</h3>
+<p>To support the notion that a user can provide hints to the replication model, we propose to add two variables to the current model.</p>
+<p>In the new model, we propose to break the <code>currentCount</code> into the two separate groups. That is <em>Healthy nodes</em> and <em>Maintenance nodes</em>. The new model replaces the currentCount with these two separate counts. The following function captures the code that drives the logic of computing Replica counts in the new model. The later section discusses the input and output of this model very extensively.</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">/**
+</span><span style="color:#75715e"> * Calculate the number of the missing replicas.
+</span><span style="color:#75715e"> *
+</span><span style="color:#75715e"> * @return the number of the missing replicas.
+</span><span style="color:#75715e"> If it's less than zero, the container is over replicated.
+</span><span style="color:#75715e"> */</span>
+<span style="color:#66d9ef">int</span> <span style="color:#a6e22e">getReplicationCount</span><span style="color:#f92672">(</span><span style="color:#66d9ef">int</span> expectedCount<span style="color:#f92672">,</span> <span style="color:#66d9ef">int</span> healthy<span style="color:#f92672">,</span> <span style="color:#66d9ef">int</span> maintenance<span style="color:#f92672">)</span> <span style="color:#f92672">{</span>
+
+ <span style="color:#75715e">//for over replication, count only with the healthy replicas
+</span><span style="color:#75715e"></span> <span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>expectedCount <span style="color:#f92672"><=</span> healthy<span style="color:#f92672">)</span> <span style="color:#f92672">{</span>
+ <span style="color:#66d9ef">return</span> expectedCount <span style="color:#f92672">-</span> healthy<span style="color:#f92672">;</span>
+ <span style="color:#f92672">}</span>
+
+ replicaCount <span style="color:#f92672">=</span> expectedCount <span style="color:#f92672">-</span> <span style="color:#f92672">(</span>healthy <span style="color:#f92672">+</span> maintenance<span style="color:#f92672">);</span>
+
+ <span style="color:#75715e">//at least one HEALTHY replicas should be guaranteed!
+</span><span style="color:#75715e"></span> <span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>replicaCount <span style="color:#f92672">==</span> 0 <span style="color:#f92672">&&</span> healthy <span style="color:#f92672"><</span> 1<span style="color:#f92672">)</span> <span style="color:#f92672">{</span>
+ replicaCount <span style="color:#f92672">++;</span>
+ <span style="color:#f92672">}</span>
+
+ <span style="color:#75715e">//over replication is already handled. Always return with
+</span><span style="color:#75715e"></span> <span style="color:#75715e">// positive as over replication is already handled
+</span><span style="color:#75715e"></span> <span style="color:#66d9ef">return</span> Math<span style="color:#f92672">.</span><span style="color:#a6e22e">max</span><span style="color:#f92672">(</span>0<span style="color:#f92672">,</span> replicaCount<span style="color:#f92672">);</span>
+<span style="color:#f92672">}</span>
+
+</code></pre></div><p>To understand the reasing behind the two special <code>if</code> condition check the examples below.</p>
+<p>We also need to specify two end condition when the DECOMMISSIONING node can be moved to the DECOMMISSIONED state or the ENTERING_MAINTENANCE mode can be moved to the IN_MAINTENANCE state.</p>
+<p>The following conditions should be true for all the containers and all the containers on the specific node should be closed.</p>
+<p><strong>From DECOMMISSIONING to DECOMMISSIONED</strong>:</p>
+<ul>
+<li>There are at least one healthy replica</li>
+<li>We have three replicas (both healthy and maintenance)</li>
+</ul>
+<p>Which means that our stop condition can be formalized as:</p>
+<pre tabindex="0"><code>(healthy >= 1) && (healthy + maintenance >= 3)
+</code></pre><p>Both the numbers can be configurable:</p>
+<ul>
+<li>1 is the minimum number of healthy replicas (<code>decommissioning.minimum.healthy-replicas</code>)</li>
+<li>3 is the minimum number of existing replicas (<code>decommissioning.minimum.replicas</code>)</li>
+</ul>
+<p>For example <code>decommissioning.minimum.healthy-replicas</code> can be set to two if administrator would like to survive an additional node failure during the maintenance period.</p>
+<p><strong>From ENTERING_MAINTENANCE to IN_MAINTENANCE:</strong></p>
+<ul>
+<li>There are at least one healthy replicas</li>
+</ul>
+<p>This is the weaker version of the previous condition:</p>
+<pre tabindex="0"><code>(healthy >= 1)
+</code></pre><h3 id="examples-normal-cases">Examples (normal cases)</h3>
+<p>In this section we show example use cases together with the output of the proposed algorithm</p>
+<h4 id="all-healthy">All healthy</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>3</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>The container C1 exists on machines A, B , and C. All the container reports tell us that the container is healthy. Running the above algorithm, we get:</p>
+<p><code>expected - healthy + maint. = 3 - (3 + 0) = 0</code></p>
+<p>It means, <em>“we don’t need no replication”.</em></p>
+<h4 id="one-failure">One failure</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>DEAD</td>
+<td>3</td>
+<td>2</td>
+<td>0</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p>The machine C has failed, and as a result, the healthy count has gone down from <code>3</code> to <code>2</code>. This means that we need to start one replication flow.</p>
+<p><code>ReplicaCount = expected - healthy + maint. = 3 - (2 + 0) = 1.</code></p>
+<p>This means that the new model will handle failure cases just like the current model.</p>
+<h4 id="one-decommissioning">One decommissioning</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>2</td>
+<td>0</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p>In this case, machine C is being decommissioned. Therefore the healthy count has gone down to <code>2</code> , and decommission count is <code>1</code>. Since the</p>
+<p><code>ReplicaCount = expected - healthy + maint`. we have `1 = 3 - (2 + 0)</code>,</p>
+<p>this gives us the decommission count implicitly. The trick here is to realize that incrementing decommission automatically causes a decrement in the healthy count, which allows us not to have <em>decommission</em> in the equation explicitly.</p>
+<p><strong>Stop condition</strong>: Not that if this containers is the only one on node C, node C can be moved to the DECOMMISSIONED state.</p>
+<h4 id="failure--decommissioning">Failure + decommissioning</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DEAD</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>1</td>
+<td>0</td>
+<td>2</td>
+</tr>
+</tbody>
+</table>
+<p>Here is a case where we have a failure of a data node and a decommission of another data node. In this case, the container C1 needs two replica flows to heal itself. The equation is the same and we get</p>
+<p><code>ReplicaCount(2) = ExpectecCount(3) - healthy(1)</code></p>
+<p>The maintenance is still zero so ignored in this equation.</p>
+<h4 id="1-failure--2-decommissioning">1 failure + 2 decommissioning</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DECOMM</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>In this case, we have one failed data node and two data nodes being decommissioned. We need to get three replica flows in the system. This is achieved by:</p>
+<pre tabindex="0"><code>ReplicaCount(3) = ExpectedCount(3) - (healthy(0) + maintenance(0))
+</code></pre><h4 id="maintenance-mode">Maintenance mode</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DEAD</td>
+<td>MAINT</td>
+<td>3</td>
+<td>2</td>
+<td>1</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>This represents the normal maintenance mode, where a single machine is marked as in maintenance mode. This means the following:</p>
+<pre tabindex="0"><code>ReplicaCount(0) = ExpectedCount(3) - (healthy(2) + maintenance(1)
+</code></pre><p>There are no replica flows since the user has asked us to move a single node into maintenance mode, and asked us explicitly not to worry about the single missing node.</p>
+<p><strong>Stop condition</strong>: Not that if this containers is the only one on node C, node C can be moved to the IN_MAINTENANCE state.</p>
+<h4 id="maintenance--decommissioning">Maintenance + decommissioning</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DECOMM</td>
+<td>MAINT</td>
+<td>3</td>
+<td>1</td>
+<td>1</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p><em>This is a fascinating case</em>; We have one good node; one decommissioned node and one node in maintenance mode. The expected result is that the replica manager will launch one replication flow to compensate for the node that is being decommissioned, and we also expect that there will be no replication for the node in maintenance mode.</p>
+<pre tabindex="0"><code>Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1))
+</code></pre><p>So as expected we have one replication flow in the system.</p>
+<p><strong>Stop condition</strong>: Not that if this containers is the only one in the system:</p>
+<ul>
+<li>node C can be moved to the IN_MAINTENANCE state</li>
+<li>node B can not be decommissioned (we need the three replicas first)</li>
+</ul>
+<h4 id="decommissioning-all-the-replicas">Decommissioning all the replicas</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DECOMM</td>
+<td>DECOMM</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>In this case, we deal with all the data nodes being decommissioned. The number of healthy replicas for this container is 0, and hence:</p>
+<pre tabindex="0"><code>replicaCount (3) = expectedCount (3)- (healthy(0) + maintenance(0)).
+</code></pre><p>This provides us with all 3 independent replica flows in the system.</p>
+<h4 id="decommissioning-the-one-remaining-replicas">Decommissioning the one remaining replicas</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DEAD</td>
+<td>DEAD</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>We have two failed nodes and one node in Decomm. It is the opposite of case Line 5, where we have one failed node and 2 nodes in Decomm. The expected results are the same, we get 3 flows.</p>
+<h4 id="total-failure">Total failure</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DEAD</td>
+<td>DEAD</td>
+<td>DEAD</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>This is really an error condition. We have lost all 3 data nodes. The Replica Manager will compute that we need to rebuild 3 replicas, but we might not have a source to rebuild from.</p>
+<h3 id="last-replica-is-on-entering_maintenance">Last replica is on ENTERING_MAINTENANCE</h3>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DEAD</td>
+<td>MAINT</td>
+<td>DEAD</td>
+<td>3</td>
+<td>0</td>
+<td>1</td>
+<td>2</td>
+</tr>
+</tbody>
+</table>
+<p>Is also an interesting case; we have lost 2 data nodes; and one node is being marked as Maint. Since we have 2 failed nodes, we need 2 replica flows in the system. However, the maintenance mode cannot be entered, since we will lose lone replica if we do that.</p>
+<h3 id="all-maintenance">All maintenance</h3>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>MAINT</td>
+<td>MAINT</td>
+<td>MAINT</td>
+<td>3</td>
+<td>0</td>
+<td>3</td>
+<td><em>1</em></td>
+</tr>
+</tbody>
+</table>
+<p>This is also a very special case; this is the case where the user is telling us to ignore the peril for all 3 replicas being offline. This means that the system will not be able to get to that container and would lead to potential I/O errors. Ozone will strive to avoid that case; this means that Ozone will hit the “if condition” and discover that we our ReplicCount is 0; since the user asked for it; but we are also going to lose all Replicas. At this point of time, we make a conscious [...]
+<p><strong>This brings us back to the semantics of Maintenance mode in Ozone</strong>. If going into maintenance mode will not lead to a potential I/O failure, we will enter into the maintenance mode; Otherwise, we will replicate and enter into the maintenance mode after the replication is done. This is just the core replication algorithm, not the complete Decommission or Maintenance mode algorithms, just how the replica manager would behave. Once we define the behavior of Replica Manag [...]
+<p>Note: this is the case why we need the seconf <code>if</code> in the model (numbers in the brackets shows the actual value):</p>
+<pre tabindex="0"><code> replicaCount(0) = expectedCount(3) - ( healthy(0) + maintenance(0) );
+
+
+ //at least one HEALTHY replicas should be guaranteed!
+ if (replicaCount(0) == 0 && healthy(0) < 1) {
+ replicaCount ++;
+ }
+</code></pre><h3 id="over-replication">Over replication</h3>
+<p>For over-replicated containers Ozone prefers to keep the replicas on the healthy nodes. We delete containers only if we have enough replicas on <em>healthy</em> nodes.</p>
+<pre tabindex="0"><code>int getReplicationCount(int expectedCount, int healthy, int maintenance) {
+
+ //for over replication, count only with the healthy replicas
+ if (expectedCount <= healthy) {
+ return expectedCount - healthy;
+ }
+
+ replicaCount = ... //calculate missing replicas
+
+ //over replication is already handled. Always return with
+ // positive as over replication is already handled
+ return Math.max(0, replicaCount);
+}
+</code></pre><p>Please note that we always assume that the the in-flight deletion are applied and the container is already deleted.</p>
+<p>There is a very rare case where the in-flight deletion is timed out (and as a result replication manager would assume the container is not deleted) BUT in the mean-time the container finally deleted. It can be survived with including the creation timestamp in the ContainerDeleteCommand.</p>
+<h3 id="over-replication-examples">Over replication examples</h3>
+<h4 id="4-replicas">4 replicas</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>Node D</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>3</td>
+<td>4</td>
+<td>0</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>This is an easy case as we have too many replicas we can safely remove on.</p>
+<pre tabindex="0"><code>if (expectedCount <= healthy) {
+ return expectedCount - healthy
+}
+</code></pre><h4 id="over-replicated-with-in_maintenance">over replicated with IN_MAINTENANCE</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>Node D</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>MAINT</td>
+<td>3</td>
+<td>3</td>
+<td>1</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>In this case we will delete the forth replica only after node D is restored and healthy again. (expectedCount is not less than healthy). As the <code>expectedCount (3) <= healthy (3)</code> the replicaCount will be calculated as <code>0</code>.</p>
+<h4 id="over-replicated-with-in_maintenance-1">over replicated with IN_MAINTENANCE</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>Node D</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>MAINT</td>
+<td>MAINT</td>
+<td>3</td>
+<td>2</td>
+<td>2</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>Here we are not over-repliacated as we don’t have any healthy nodes. We will calculate the under-replication number as defined in the previous section:</p>
+<pre tabindex="0"><code>replicaCount(-1) = expectedCount(3) - ( healthy(2) + maintenance(2) );
+</code></pre><p>The main algorithm would return with <code>replicaCount = -1</code> but as we return <code>Math.max(0,replicaCount)</code> the real response will be 0. Waiting for healthy nodes.</p>
+<h3 id="handling-in-flight-replications">Handling in-flight replications</h3>
+<p>In-flight replication requests and deletes are handled by the Replica Manager and the problem is orthogonal to the replication problem, but this section shows that the proposed model is not conflicted with the existing approach.</p>
+<p>Let’s say we have an under-replicated container and we already selected a new datanode to copy a new replica to that specific node.</p>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>(copy)</td>
+<td>3</td>
+<td>2</td>
+<td>0</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p>Here the Replication Manager detects that one replica is missing but the real copy shouldn’t been requested as it’s alread inprogress. ReplicaManager must not select a new datanode based on the ContainerPlacementPolicy implementation as the policy may or may not be idempotent.</p>
+<p>For example if the placement policy would select a datanode randomly with each loop we would select a new datanode to replicate to.</p>
+<p>To avoid such a situation Replica Manager maintains a list of the in-flight copies (in-memory) on the SCM side. In this list we have all the sent replication requests but they are removed after a given amount of time (10 minutes) by default.</p>
+<p>With calculating the in-flight copy as a possible replication the Replication Manger doesn’t need to request new replication.</p>
+<p>When a datanode is marked to be decommissioned there could be any in-flight replication copy process in that time.</p>
+<ul>
+<li>At datanode we should stop all of the in-flight copy (datanodes should be notified about the DECOMMISSIONING/IN_MAINTENANCE state)</li>
+<li>We never ask any non-healthy nodes to replicate containers.</li>
+<li>In SCM, we don’t need to do any special action
+<ul>
+<li>In <code>ReplicationManager</code> we already have a map about the inflight replications (<code>Map<ContainerID, List<InflightAction>></code>).</li>
+<li>During a normal replication the number of in-flight replications are counted as real replication (2 real replicas + 1 inflight replication = replica count 3). During this calculation we need to check the current state of the datanodes and ignore the inflight replication if they are assigned to a node which is in decommissioning state. (Or we should update the inflight map, in case of node state change)</li>
+</ul>
+</li>
+</ul>
+<h3 id="in-flight-examples">In-flight examples</h3>
+<h4 id="maintenance--inflight">Maintenance + inflight</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+<th>copy</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>MAINT</td>
+<td>copying</td>
+<td>3</td>
+<td>1</td>
+<td>1</td>
+<td>1</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p>Here one have one node ENTERING_MAINTENANCE state, and one replica is missing and already started to be replicated. We don’t need to start a new copy and node B can be moved to the IN_MAINTENANCE mode.</p>
+<pre tabindex="0"><code>Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1))
+Containers to copy (0) = Replica Count (1) - inflight copies (1)
+</code></pre><h4 id="maintenance--inflight-1">Maintenance + inflight</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+<th>copy</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DECOMM</td>
+<td>copying</td>
+<td>copying</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p>Node A can not be DECOMMISSIONED as we have no HEALTHY replicas at all.</p>
+<h2 id="statefulness">Statefulness</h2>
+<p>SCM stores all the node state in-memory. After a restart on the SCM side the datanode state can be lost.</p>
+<p><strong>Ozone doesn’t guarantee that decommissioning/maintenance mode state survives the SCM restarts!!!</strong></p>
+<ul>
+<li>If SCM restarts DECOMMISSIONED nodes will not report any more container reports and the nodes won’t be registered.</li>
+<li>ENTERING_MAINTENANCE and DECOMMISSIONING nodes will became HEALTHY again and the decommissioning CLI command should be repeated.</li>
+<li>IN_MAINTENANCE nodes will become DEAD and all the containers will be replicated.</li>
+</ul>
+<p><em>Ozone assumes that the maintenance mode is used short-term and SCM is not restarted during this specific period.</em></p>
+<p><em>Reasoning</em>:</p>
+<p>Neither of the node state nor the container state are persisted in SCM side. The decommissioned state can be stored on the SCM side (or on the SCM side and the datanode side) which can provide better user experience (and may be implemented).</p>
+<p>But to support maintenance mode after restart all the container information is required to be persisted (which is a too big architectural change).</p>
+<p>To make a replication decision replication manager needs the number of healthy replicas (they are reported via heartbeats) AND the number of containers on the node which is in maintenance mode. The later one is not available if the SCM is restarted as the container map exists only in the memory and the node which is turned off can’t report any more container reports. Therefore the information about the existing containers on the node which is in the maintenance mode <strong>can& [...]
+<h2 id="throttling">Throttling</h2>
+<p>SCM should avoid to request too many replication to live enough network bandwidth for the requests.</p>
+<p>Replication Manager can easily throttle the replication requests based on <code>inflightReplication</code> map, but this problem is independent from the handling of the decommissioning / maintenance mode because it should be solved for any kind of replication not just for this.</p>
+<h2 id="user-interface">User interface</h2>
+<p>The decommissioning and maintenance mode can be administered with a CLI interface.</p>
+<p>Required feature:</p>
+<ul>
+<li>Set the state of a datanode (to DECOMMISSIONING or ENTERING_MAINTENANCE)</li>
+<li>Undo the decommissioning process</li>
+<li>check the current progress:
+<ul>
+<li>This can be a table with the nodes, status of the nodes, number of containers, containers under replication and containers which doesn’t much the stop condition yet (required replications)</li>
+</ul>
+</li>
+<li>All the commands can support topology related filters (eg. display the nodes only for a specific rack or show the status of the nodes of s specific rack)</li>
+<li>Check current replication information of one specific container (required to debug why the decommissioning is stuck)</li>
+</ul>
+<h2 id="checks-before-the-decommissioning">Checks before the decommissioning</h2>
+<p>Decommissioning is requested via a new RPC call with the help of a new CLI tool. The server should check the current state of the cluster and deny the decommissioning if it’s not possible. Possible violations:</p>
+<ul>
+<li>Not enough space to store the new replicas.</li>
+<li>Not enough node to create all kind of pipelines</li>
+</ul>
+<p>In case of any violation, the request will fail, but any of theses rules can be turned off with a next request and the decommissioning can be forced.</p>
+<h2 id="maintain-progress">Maintain progress</h2>
+<p>We need to show the progress of the decommissioning process per node and cluster-wide. We already have the information about the under replicated containers, but we don’t know the numbers of the containers before decommissioning.</p>
+<p>Instead of saving the original number of the required replications before (which is very fragile) we don’t provide an absolute progress just the numbers of the remaining replication:</p>
+<pre tabindex="0"><code> Node | Status | # containers | in-progress replications| required replication
+ ----------------|------------------------|--------------|--------------------------|------------------------
+ Node A | ENTERING_MAINTENANCE | 2837 | 12 | 402
+ Node B | HEALTHY | 1239 | 0 | 0
+ Node C | IN_MAINTENANCE | 2348 | 0 | 0
+</code></pre><p><code># containers</code> means the total number of the containers on the specific datanodes. To get the original number of the planned copies we can save the original ‘container-to-node’ map in memory and show some progress and provide more information for the users.</p>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: March 9, 2021 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/45eee3fa5d29f49eaed0758d1703d446d56e70fb">45eee3fa5</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/ec.html b/docs/1.2.0/design/ec.html
new file mode 100644
index 0000000..94cbe2c
--- /dev/null
+++ b/docs/1.2.0/design/ec.html
@@ -0,0 +1,402 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-3816">[HDDS-3816]</a> Erasure Coding in Ozone (draft) </h1>
+ <div><i>Authors: Uma Maheswara Rao Gangumalla, Marton Elek, Stephen O'Donnell</i><div class="pull-right">2020-06-30</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Use Erasure Coding algorithm for efficient storage
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>Support Erasure Coding for read and write pipeline of Ozone.</p>
+<h1 id="status">Status</h1>
+<p>The design doc describes two main methods to implement EC:</p>
+<ul>
+<li>Container level, async Erasure Coding, to encode closed containers in the background</li>
+<li>Block level, striped Erasure Coding</li>
+</ul>
+<p>Second option can work only with new, dedicated write-path. Details of possible implementation will be included in the next version.</p>
+<h1 id="link">Link</h1>
+<p><a href="https://issues.apache.org/jira/secure/attachment/13006245/Erasure%20Coding%20in%20Apache%20Hadoop%20Ozone.pdf">https://issues.apache.org/jira/secure/attachment/13006245/Erasure%20Coding%20in%20Apache%20Hadoop%20Ozone.pdf</a></p>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: August 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/5ce6f0eab381389fd04c3130531b3ec626acbc65">5ce6f0eab</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/gdpr.html b/docs/1.2.0/design/gdpr.html
new file mode 100644
index 0000000..9dcd6cc
--- /dev/null
+++ b/docs/1.2.0/design/gdpr.html
@@ -0,0 +1,396 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-2012">[HDDS-2012]</a> Ozone GDPR framework (implemented) </h1>
+ <div><i>Authors: Dinesh Chitlangia</i><div class="pull-right">2019-06-20</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Crypto key management to handle GDPR “right to be forgotten” feature
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>Encrypt all the key and delete only the encryption key if the deletion is requested. Without the key any remaining data can’t be read any more.</p>
+<h1 id="link">Link</h1>
+<p><a href="https://issues.apache.org/jira/secure/attachment/12978992/Ozone%20GDPR%20Framework.pdf">https://issues.apache.org/jira/secure/attachment/12978992/Ozone%20GDPR%20Framework.pdf</a></p>
+<p><a href="https://issues.apache.org/jira/secure/attachment/12987528/Ozone%20GDPR%20Framework_updated.pdf">https://issues.apache.org/jira/secure/attachment/12987528/Ozone%20GDPR%20Framework_updated.pdf</a></p>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: April 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/9872253e89377624dfea5fb9b321538fc70f91c4">9872253e8</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/index.xml b/docs/1.2.0/design/index.xml
new file mode 100644
index 0000000..7e98ae2
--- /dev/null
+++ b/docs/1.2.0/design/index.xml
@@ -0,0 +1,245 @@
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
+<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
+ <channel>
+ <title>Designs on Ozone</title>
+ <link>/design.html</link>
+ <description>Recent content in Designs on Ozone</description>
+ <generator>Hugo -- gohugo.io</generator>
+ <language>en-us</language>
+ <lastBuildDate>Mon, 12 Apr 2021 00:00:00 +0000</lastBuildDate><atom:link href="/design/index.xml" rel="self" type="application/rss+xml" />
+ <item>
+ <title>Ozone FS namespace / prefix table</title>
+ <link>/design/namespace-support.html</link>
+ <pubDate>Mon, 12 Apr 2021 00:00:00 +0000</pubDate>
+
+ <guid>/design/namespace-support.html</guid>
+ <description>Use additional prefix table for indexed data retrieval</description>
+ </item>
+
+ <item>
+ <title>Non Rolling upgrade support in Ozone.</title>
+ <link>/design/nonrolling-upgrade.html</link>
+ <pubDate>Mon, 15 Feb 2021 00:00:00 +0000</pubDate>
+
+ <guid>/design/nonrolling-upgrade.html</guid>
+ <description>‘Non rolling’ upgrade support in Ozone.</description>
+ </item>
+
+ <item>
+ <title>Ozone Manager Prepare for Upgrade</title>
+ <link>/design/omprepare.html</link>
+ <pubDate>Mon, 15 Feb 2021 00:00:00 +0000</pubDate>
+
+ <guid>/design/omprepare.html</guid>
+ <description>‘Prepare upgrade’ step to make sure all the OMs use the same version of the software to update their DBs (apply transaction) for a given request.</description>
+ </item>
+
+ <item>
+ <title>Upgrade - Developer Primer</title>
+ <link>/design/upgrade-dev-primer.html</link>
+ <pubDate>Mon, 15 Feb 2021 00:00:00 +0000</pubDate>
+
+ <guid>/design/upgrade-dev-primer.html</guid>
+ <description>Helpful resources for those who are bringing layout changes.</description>
+ </item>
+
+ <item>
+ <title>Proposed persistent OM connection for S3 gateway</title>
+ <link>/design/s3-performance.html</link>
+ <pubDate>Mon, 09 Nov 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/s3-performance.html</guid>
+ <description>Proposal to use per-request authentication and persistent connections between S3g and OM</description>
+ </item>
+
+ <item>
+ <title>Erasure Coding in Ozone</title>
+ <link>/design/ec.html</link>
+ <pubDate>Tue, 30 Jun 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/ec.html</guid>
+ <description>Use Erasure Coding algorithm for efficient storage</description>
+ </item>
+
+ <item>
+ <title>Implement new Ozone FileSystem scheme ofs://</title>
+ <link>/design/ofs.html</link>
+ <pubDate>Tue, 30 Jun 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/ofs.html</guid>
+ <description>A new schema structure for Hadoop compatible file system</description>
+ </item>
+
+ <item>
+ <title>Storage Class</title>
+ <link>/design/storage-class.html</link>
+ <pubDate>Mon, 08 Jun 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/storage-class.html</guid>
+ <description>New abstraction to configure replication methods.</description>
+ </item>
+
+ <item>
+ <title>Ozone Volume Management</title>
+ <link>/design/volume-management.html</link>
+ <pubDate>Thu, 02 Apr 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/volume-management.html</guid>
+ <description>A simplified version of mapping between S3 buckets and Ozone volume/buckets</description>
+ </item>
+
+ <item>
+ <title>NFS support Ozone</title>
+ <link>/design/nfs.html</link>
+ <pubDate>Wed, 25 Mar 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/nfs.html</guid>
+ <description>Adopt NFS gateway of HDFS and provide NFS file system view</description>
+ </item>
+
+ <item>
+ <title>SCM HA support</title>
+ <link>/design/scmha.html</link>
+ <pubDate>Thu, 05 Mar 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/scmha.html</guid>
+ <description>HA for Storage Container Manager using Ratis to replicate data</description>
+ </item>
+
+ <item>
+ <title>Generic Extensible Token support for Ozone</title>
+ <link>/design/token.html</link>
+ <pubDate>Thu, 09 Jan 2020 00:00:00 +0000</pubDate>
+
+ <guid>/design/token.html</guid>
+ <description>Extend existing token architecture to all entities of Ozone</description>
+ </item>
+
+ <item>
+ <title>Ozone Trash Feature</title>
+ <link>/design/trash.html</link>
+ <pubDate>Thu, 07 Nov 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/trash.html</guid>
+ <description>Feature provides a user with the ability to recover keys that may have been deleted accidentally. (similar to the HDFS trash feature).</description>
+ </item>
+
+ <item>
+ <title>Recon server v2</title>
+ <link>/design/recon2.html</link>
+ <pubDate>Mon, 19 Aug 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/recon2.html</guid>
+ <description>Second phase of Recon development. Recon acts as a passive SCM.</description>
+ </item>
+
+ <item>
+ <title>Decommissioning in Ozone</title>
+ <link>/design/decommissioning.html</link>
+ <pubDate>Wed, 31 Jul 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/decommissioning.html</guid>
+ <description>Formal process to shut down machines in a safe way after the required replications.</description>
+ </item>
+
+ <item>
+ <title>Ozone GDPR framework</title>
+ <link>/design/gdpr.html</link>
+ <pubDate>Thu, 20 Jun 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/gdpr.html</guid>
+ <description>Crypto key management to handle GDPR &ldquo;right to be forgotten&rdquo; feature</description>
+ </item>
+
+ <item>
+ <title>Ozone locks in OM</title>
+ <link>/design/locks.html</link>
+ <pubDate>Thu, 20 Jun 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/locks.html</guid>
+ <description>Hierarchical locking structure for OM elements (volumes/keys/buckets)</description>
+ </item>
+
+ <item>
+ <title>Ozone Enhancement Proposals</title>
+ <link>/design/ozone-enhancement-proposals.html</link>
+ <pubDate>Fri, 07 Jun 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/ozone-enhancement-proposals.html</guid>
+ <description>Definition of the process to share new technical proposals with the Ozone community.</description>
+ </item>
+
+ <item>
+ <title>Configless Ozone service management</title>
+ <link>/design/configless.html</link>
+ <pubDate>Sat, 25 May 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/configless.html</guid>
+ <description>Distribute only minimal configuration and download all the remaining before start</description>
+ </item>
+
+ <item>
+ <title>Ozone multi-raft support</title>
+ <link>/design/multiraft.html</link>
+ <pubDate>Tue, 21 May 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/multiraft.html</guid>
+ <description>Datanodes can be part of multiple independent RAFT groups / pipelines</description>
+ </item>
+
+ <item>
+ <title>Type-safe configuration API</title>
+ <link>/design/typesafeconfig.html</link>
+ <pubDate>Thu, 25 Apr 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/typesafeconfig.html</guid>
+ <description>Inject configuration values based on annotations instead of using constants and Hadoop API</description>
+ </item>
+
+ <item>
+ <title>Recon server 1</title>
+ <link>/design/recon1.html</link>
+ <pubDate>Tue, 19 Feb 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/recon1.html</guid>
+ <description>Second phase of Recon development. Recon acts as a passive SCM.</description>
+ </item>
+
+ <item>
+ <title>Ozone Encryption At Rest</title>
+ <link>/design/tde.html</link>
+ <pubDate>Fri, 08 Feb 2019 00:00:00 +0000</pubDate>
+
+ <guid>/design/tde.html</guid>
+ <description>Transparent Data Encryption encrypts the data before store it</description>
+ </item>
+
+ <item>
+ <title>Topology-awareness</title>
+ <link>/design/topology.html</link>
+ <pubDate>Fri, 16 Nov 2018 00:00:00 +0000</pubDate>
+
+ <guid>/design/topology.html</guid>
+ <description>Placement policy to use rack information for read and write</description>
+ </item>
+
+ <item>
+ <title>S3 protocol support for Ozone</title>
+ <link>/design/s3gateway.html</link>
+ <pubDate>Thu, 27 Sep 2018 00:00:00 +0000</pubDate>
+
+ <guid>/design/s3gateway.html</guid>
+ <description>Support any AWS S3 compatible client with dedicated REST endpoint</description>
+ </item>
+
+ <item>
+ <title>Ozone Manager HA</title>
+ <link>/design/omha.html</link>
+ <pubDate>Tue, 18 Sep 2018 00:00:00 +0000</pubDate>
+
+ <guid>/design/omha.html</guid>
+ <description>Support HA for Ozone Manager with the help of RATIS</description>
+ </item>
+
+ </channel>
+</rss>
diff --git a/docs/1.2.0/design/locks.html b/docs/1.2.0/design/locks.html
new file mode 100644
index 0000000..31e6e87
--- /dev/null
+++ b/docs/1.2.0/design/locks.html
@@ -0,0 +1,395 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-1672">[HDDS-1672]</a> Ozone locks in OM (implemented) </h1>
+ <div><i>Authors: Bharat Viswanadham</i><div class="pull-right">2019-06-20</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Hierarchical locking structure for OM elements (volumes/keys/buckets)
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>Fine grained locking hierarchy for Ozone specific resources (volumes, buckets, keys…)</p>
+<h1 id="link">Link</h1>
+<p><a href="https://issues.apache.org/jira/secure/attachment/12972275/Ozone%20Locks%20in%20OM.pdf">https://issues.apache.org/jira/secure/attachment/12972275/Ozone%20Locks%20in%20OM.pdf</a></p>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: April 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/9872253e89377624dfea5fb9b321538fc70f91c4">9872253e8</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/multiraft.html b/docs/1.2.0/design/multiraft.html
new file mode 100644
index 0000000..e079a2d
--- /dev/null
+++ b/docs/1.2.0/design/multiraft.html
@@ -0,0 +1,396 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-1564">[HDDS-1564]</a> Ozone multi-raft support (implemented) </h1>
+ <div><i>Authors: Li Cheng, Sammi Chen</i><div class="pull-right">2019-05-21</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Datanodes can be part of multiple independent RAFT groups / pipelines
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>Without multiraft support one datanode could be part of one RAFT group. With multiraft one datanode can be part of multiple independent RAFT ring.</p>
+<h1 id="link">Link</h1>
+<p><a href="https://issues.apache.org/jira/secure/attachment/12990694/multiraft_performance_brief.pdf">https://issues.apache.org/jira/secure/attachment/12990694/multiraft_performance_brief.pdf</a></p>
+<p><a href="https://issues.apache.org/jira/secure/attachment/12969227/Ozone%20Multi-Raft%20Support.pdf">https://issues.apache.org/jira/secure/attachment/12969227/Ozone%20Multi-Raft%20Support.pdf</a></p>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: July 27, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/e643ab2761f66bd223a73242c530e29c686244d3">e643ab276</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/namespace-support.html b/docs/1.2.0/design/namespace-support.html
new file mode 100644
index 0000000..36b940f
--- /dev/null
+++ b/docs/1.2.0/design/namespace-support.html
@@ -0,0 +1,398 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-2939">[HDDS-2939]</a> Ozone FS namespace / prefix table (implemented) </h1>
+ <div><i>Authors: Supratim Deka, Anu Engineer, Rakesh Radhakrishnan</i><div class="pull-right">2021-04-12</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Use additional prefix table for indexed data retrieval
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>Flat namespace (like key -> key info) is not efficient for listing/deleting/renaming directories. (Large segments should be scanned, the whole sub-hierarchy). To make deletion / rename fast and atomic (and make the lists faster) the key table is separated for prefix + key table.</p>
+<h1 id="link">Link</h1>
+<ul>
+<li><a href="https://issues.apache.org/jira/secure/attachment/12991926/Ozone%20FS%20Namespace%20Proposal%20v1.0.docx">Design doc</a></li>
+<li><a href="https://issues.apache.org/jira/secure/attachment/13023399/OzoneFS%20Optimizations_DesignOverview_%20HDDS-2939.pdf">Quick overview</a></li>
+</ul>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: April 12, 2021 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/ec38f01de82ecb6dab4294924e4602fe87ee6c06">ec38f01de</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/nfs.html b/docs/1.2.0/design/nfs.html
new file mode 100644
index 0000000..ab05ea3
--- /dev/null
+++ b/docs/1.2.0/design/nfs.html
@@ -0,0 +1,401 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-3001">[HDDS-3001]</a> NFS support Ozone (draft) </h1>
+ <div><i>Authors: Prashant Pogde</i><div class="pull-right">2020-03-25</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ Adopt NFS gateway of HDFS and provide NFS file system view
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<ol>
+<li>Use NFS server of HDFS and adopt it</li>
+<li>Support unique ID for each Ozone object</li>
+<li>Support Random Write</li>
+</ol>
+<h1 id="link">Link</h1>
+<ul>
+<li><a href="https://issues.apache.org/jira/secure/attachment/12997790/NFS%20Support%20for%20Ozone.pdf">https://issues.apache.org/jira/secure/attachment/12997790/NFS%20Support%20for%20Ozone.pdf</a></li>
+</ul>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: April 7, 2020 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/9872253e89377624dfea5fb9b321538fc70f91c4">9872253e8</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/nonrolling-upgrade.html b/docs/1.2.0/design/nonrolling-upgrade.html
new file mode 100644
index 0000000..6695ee9
--- /dev/null
+++ b/docs/1.2.0/design/nonrolling-upgrade.html
@@ -0,0 +1,397 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <a href="../index.html" class="navbar-left ozone-logo">
+ <img src="../ozone-logo-small.png"/>
+ </a>
+ <a class="navbar-brand hidden-xs" href="../index.html">
+ Apache Ozone/HDDS documentation
+ </a>
+ <a class="navbar-brand visible-xs-inline" href="#">Apache Ozone</a>
+ </div>
+ <div id="navbar" class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="https://github.com/apache/hadoop-ozone">Source</a></li>
+ <li><a href="https://hadoop.apache.org">Apache Hadoop</a></li>
+ <li><a href="https://apache.org">ASF</a></li>
+ </ul>
+ </div>
+ </div>
+</nav>
+
+ <div class="container-fluid">
+ <div class="row">
+
+<div class="col-sm-2 col-md-2 sidebar" id="sidebar">
+ <ul class="nav nav-sidebar">
+
+
+
+ <li class="">
+
+ <a href="../index.html">
+
+
+
+ <span>Overview</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../start.html">
+
+
+
+ <span>Getting Started</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+ <a href="../concept.html">
+
+ <span>Architecture</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../concept/overview.html">Overview</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/ozonemanager.html">Ozone Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/storagecontainermanager.html">Storage Container Manager</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/containers.html">Containers</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/datanodes.html">Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../concept/recon.html">Recon</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../feature.html">
+
+ <span>Features</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../feature/om-ha.html">OM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/scm-ha.html">SCM High Availability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/prefixfso.html">Prefix based FileSystem Optimization</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/topology.html">Topology awareness</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/quota.html">Quota in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/recon.html">Recon Server</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/observability.html">Observability</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../feature/nonrolling-upgrade.html">Non-Rolling Upgrades and Downgrades</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../interface.html">
+
+ <span>Client Interfaces</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../interface/ofs.html">Ofs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/o3fs.html">O3fs (Hadoop compatible)</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/s3.html">S3 Protocol</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/cli.html">Command Line Interface</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/reconapi.html">Recon API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/javaapi.html">Java API</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../interface/csi.html">CSI Protocol</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+ <a href="../security.html">
+
+ <span>Security</span>
+ </a>
+ <ul class="nav">
+
+ <li class="">
+
+ <a href="../security/secureozone.html">Securing Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingtde.html">Transparent Data Encryption</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/gdpr.html">GDPR in Ozone</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingdatanodes.html">Securing Datanodes</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securingozonehttp.html">Securing HTTP</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securings3.html">Securing S3</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securityacls.html">Ozone ACLs</a>
+
+ </li>
+
+ <li class="">
+
+ <a href="../security/securitywithranger.html">Apache Ranger</a>
+
+ </li>
+
+ </ul>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../tools.html">
+
+
+
+ <span>Tools</span>
+ </a>
+ </li>
+
+
+
+ <li class="">
+
+ <a href="../recipe.html">
+
+
+
+ <span>Recipes</span>
+ </a>
+ </li>
+
+
+ <li><a href="../design.html"><span><b>Design docs</b></span></a></li>
+ <li class="visible-xs"><a href="#">References</a>
+ <ul class="nav">
+ <li><a href="https://github.com/apache/hadoop"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Source</a></li>
+ <li><a href="https://hadoop.apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> Apache Hadoop</a></li>
+ <li><a href="https://apache.org"><span class="glyphicon glyphicon-new-window" aria-hidden="true"></span> ASF</a></li>
+ </ul></li>
+ </ul>
+
+</div>
+
+ <div class="col-sm-9 col-sm-offset-3 col-md-10 col-md-offset-2 main">
+ <div class="col-md-9">
+ <h1><a href="https://issues.apache.org/jira/browse/HDDS-3698">[HDDS-3698]</a> Non Rolling upgrade support in Ozone. (implemented) </h1>
+ <div><i>Authors: Aravindan Vijayan</i><div class="pull-right">2021-02-15</div></div>
+ <p> </p>
+
+ <div class="panel panel-success">
+ <div class="panel-heading">Summary</div>
+ <div class="panel-body">
+ ‘Non rolling’ upgrade support in Ozone.
+ </div>
+ </div>
+
+ <!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+<h1 id="abstract">Abstract</h1>
+<p>A layout version management based framework for supporting non rolling
+upgrades in Ozone.</p>
+<h1 id="link">Link</h1>
+<p><a href="https://issues.apache.org/jira/secure/attachment/13023446/Ozone%20Non-Rolling%20Upgrades%20Doc%20v1.2%20%28Implemented%20Design%29.pdf">https://issues.apache.org/jira/secure/attachment/13023446/Ozone%20Non-Rolling%20Upgrades%20Doc%20v1.2%20%28Implemented%20Design%29.pdf</a></p>
+<p><a href="https://issues.apache.org/jira/secure/attachment/13005731/Ozone%20Non-Rolling%20Upgrades%20%28Presentation%29.pdf">https://issues.apache.org/jira/secure/attachment/13005731/Ozone%20Non-Rolling%20Upgrades%20%28Presentation%29.pdf</a></p>
+
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+
+<footer class="footer">
+ <div class="container">
+ <span class="small text-muted">
+ Version: 1.2.0, Last Modified: April 6, 2021 <a class="hide-child link primary-color" href="https://github.com/apache/ozone/commit/3ea88e1403f957297435e238d1c8b57f0ea2304d">3ea88e140</a>
+ </span>
+ </div>
+</footer>
+
+
+
+<script src="../js/jquery-3.5.1.min.js"></script>
+<script src="../js/ozonedoc.js"></script>
+<script src="../js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
diff --git a/docs/1.2.0/design/ofs.html b/docs/1.2.0/design/ofs.html
new file mode 100644
index 0000000..836461e
--- /dev/null
+++ b/docs/1.2.0/design/ofs.html
@@ -0,0 +1,398 @@
+
+
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <meta name="description" content="Apache Ozone Documentation">
+
+ <title>Documentation for Apache Ozone</title>
+
+
+ <link href="../css/bootstrap.min.css" rel="stylesheet">
+
+
+ <link href="../css/ozonedoc.css" rel="stylesheet">
+
+ </head>
+
+
+ <body>
+
+<nav class="navbar navbar-inverse navbar-fixed-top">
+ <div class="container-fluid">
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#sidebar" aria-expanded="false" aria-controls="navbar">
... 53112 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org