You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by we...@apache.org on 2022/12/23 01:21:15 UTC
[ozone-site] 01/03: Embed Matomo trace code into 0.5.0-beta ~ 1.3.0 docs.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
commit 790899e58a65e47d7076eba64a69c6fba66d2e77
Author: Wei-Chiu Chuang <we...@apache.org>
AuthorDate: Thu Dec 22 16:27:45 2022 -0800
Embed Matomo trace code into 0.5.0-beta ~ 1.3.0 docs.
---
docs/0.5.0-beta/beyond.html | 23 +-
docs/0.5.0-beta/beyond/containers.html | 132 +--
docs/0.5.0-beta/beyond/dockercheatsheet.html | 69 +-
docs/0.5.0-beta/beyond/index.xml | 10 +-
docs/0.5.0-beta/beyond/runningwithhdfs.html | 49 +-
docs/0.5.0-beta/concept.html | 23 +-
docs/0.5.0-beta/concept/datanodes.html | 43 +-
docs/0.5.0-beta/concept/hdds.html | 27 +-
docs/0.5.0-beta/concept/index.xml | 10 +-
docs/0.5.0-beta/concept/overview.html | 56 +-
docs/0.5.0-beta/concept/ozonemanager.html | 74 +-
docs/0.5.0-beta/design.html | 25 +-
docs/0.5.0-beta/design/decommissioning.html | 1106 +++++++++++---------
docs/0.5.0-beta/design/index.xml | 13 +-
.../design/ozone-enhancement-proposals.html | 156 ++-
docs/0.5.0-beta/en/index.html | 11 +-
docs/0.5.0-beta/en/sitemap.xml | 267 ++---
docs/0.5.0-beta/gdpr.html | 25 +-
docs/0.5.0-beta/gdpr/gdpr-in-ozone.html | 30 +-
docs/0.5.0-beta/gdpr/index.xml | 10 +-
docs/0.5.0-beta/index.html | 37 +-
docs/0.5.0-beta/index.xml | 34 +-
docs/0.5.0-beta/interface.html | 23 +-
docs/0.5.0-beta/interface/index.xml | 11 +-
docs/0.5.0-beta/interface/javaapi.html | 191 ++--
docs/0.5.0-beta/interface/ozonefs.html | 139 ++-
docs/0.5.0-beta/interface/s3.html | 121 +--
docs/0.5.0-beta/recipe.html | 23 +-
docs/0.5.0-beta/recipe/index.xml | 10 +-
docs/0.5.0-beta/recipe/prometheus.html | 121 ++-
docs/0.5.0-beta/recipe/sparkozonefsk8s.html | 167 ++-
docs/0.5.0-beta/security.html | 25 +-
docs/0.5.0-beta/security/index.xml | 10 +-
docs/0.5.0-beta/security/secuitywithranger.html | 28 +-
docs/0.5.0-beta/security/secureozone.html | 44 +-
docs/0.5.0-beta/security/securingdatanodes.html | 53 +-
docs/0.5.0-beta/security/securings3.html | 53 +-
docs/0.5.0-beta/security/securingtde.html | 58 +-
docs/0.5.0-beta/security/securityacls.html | 35 +-
docs/0.5.0-beta/shell.html | 23 +-
docs/0.5.0-beta/shell/bucketcommands.html | 77 +-
docs/0.5.0-beta/shell/format.html | 60 +-
docs/0.5.0-beta/shell/index.xml | 10 +-
docs/0.5.0-beta/shell/keycommands.html | 99 +-
docs/0.5.0-beta/shell/volumecommands.html | 82 +-
docs/0.5.0-beta/sitemap.xml | 32 +-
docs/0.5.0-beta/start.html | 63 +-
docs/0.5.0-beta/start/fromsource.html | 54 +-
docs/0.5.0-beta/start/index.xml | 30 +-
docs/0.5.0-beta/start/kubernetes.html | 45 +-
docs/0.5.0-beta/start/minikube.html | 73 +-
docs/0.5.0-beta/start/onprem.html | 141 ++-
docs/0.5.0-beta/start/runningviadocker.html | 59 +-
docs/0.5.0-beta/start/startfromdockerhub.html | 80 +-
docs/0.5.0-beta/tools.html | 33 +-
docs/0.5.0-beta/tools/auditparser.html | 45 +-
docs/0.5.0-beta/tools/genconf.html | 24 +-
docs/0.5.0-beta/tools/index.xml | 10 +-
docs/0.5.0-beta/tools/scmcli.html | 24 +-
docs/0.5.0-beta/tools/testtools.html | 83 +-
docs/0.5.0-beta/zh/concept.html | 23 +-
docs/0.5.0-beta/zh/concept/datanodes.html | 23 +-
docs/0.5.0-beta/zh/concept/hdds.html | 27 +-
docs/0.5.0-beta/zh/concept/index.xml | 10 +-
docs/0.5.0-beta/zh/concept/overview.html | 44 +-
docs/0.5.0-beta/zh/concept/ozonemanager.html | 78 +-
docs/0.5.0-beta/zh/gdpr.html | 25 +-
docs/0.5.0-beta/zh/gdpr/gdpr-in-ozone.html | 28 +-
docs/0.5.0-beta/zh/gdpr/index.xml | 10 +-
docs/0.5.0-beta/zh/index.html | 39 +-
docs/0.5.0-beta/zh/index.xml | 51 +-
docs/0.5.0-beta/zh/interface.html | 23 +-
docs/0.5.0-beta/zh/interface/index.xml | 11 +-
docs/0.5.0-beta/zh/interface/javaapi.html | 191 ++--
docs/0.5.0-beta/zh/interface/ozonefs.html | 120 +--
docs/0.5.0-beta/zh/interface/s3.html | 123 +--
docs/0.5.0-beta/zh/shell.html | 23 +-
docs/0.5.0-beta/zh/shell/bucketcommands.html | 85 +-
docs/0.5.0-beta/zh/shell/format.html | 58 +-
docs/0.5.0-beta/zh/shell/index.xml | 10 +-
docs/0.5.0-beta/zh/shell/keycommands.html | 114 +-
docs/0.5.0-beta/zh/shell/volumecommands.html | 92 +-
docs/0.5.0-beta/zh/sitemap.xml | 131 +--
docs/0.5.0-beta/zh/start.html | 63 +-
docs/0.5.0-beta/zh/start/fromsource.html | 50 +-
docs/0.5.0-beta/zh/start/index.xml | 50 +-
docs/0.5.0-beta/zh/start/kubernetes.html | 45 +-
docs/0.5.0-beta/zh/start/minikube.html | 73 +-
docs/0.5.0-beta/zh/start/onprem.html | 138 +--
docs/0.5.0-beta/zh/start/runningviadocker.html | 58 +-
docs/0.5.0-beta/zh/start/startfromdockerhub.html | 78 +-
docs/1.0.0/concept.html | 22 +
docs/1.0.0/concept/containers.html | 22 +
docs/1.0.0/concept/datanodes.html | 22 +
docs/1.0.0/concept/index.xml | 8 +-
docs/1.0.0/concept/overview.html | 22 +
docs/1.0.0/concept/ozonemanager.html | 22 +
docs/1.0.0/concept/storagecontainermanager.html | 22 +
docs/1.0.0/design.html | 22 +
docs/1.0.0/design/configless.html | 22 +
docs/1.0.0/design/decommissioning.html | 119 ++-
docs/1.0.0/design/ec.html | 22 +
docs/1.0.0/design/gdpr.html | 22 +
docs/1.0.0/design/index.xml | 8 +-
docs/1.0.0/design/locks.html | 22 +
docs/1.0.0/design/multiraft.html | 22 +
docs/1.0.0/design/namespace-support.html | 22 +
docs/1.0.0/design/nfs.html | 22 +
docs/1.0.0/design/ofs.html | 22 +
docs/1.0.0/design/omha.html | 22 +
docs/1.0.0/design/ozone-enhancement-proposals.html | 55 +-
docs/1.0.0/design/recon1.html | 22 +
docs/1.0.0/design/recon2.html | 22 +
docs/1.0.0/design/s3gateway.html | 22 +
docs/1.0.0/design/scmha.html | 22 +
docs/1.0.0/design/storage-class.html | 22 +
docs/1.0.0/design/tde.html | 22 +
docs/1.0.0/design/token.html | 22 +
docs/1.0.0/design/topology.html | 22 +
docs/1.0.0/design/trash.html | 22 +
docs/1.0.0/design/typesafeconfig.html | 22 +
docs/1.0.0/design/volume-management.html | 36 +-
docs/1.0.0/en/index.html | 11 +-
docs/1.0.0/en/sitemap.xml | 368 +++----
docs/1.0.0/feature.html | 22 +
docs/1.0.0/feature/gdpr.html | 30 +-
docs/1.0.0/feature/ha.html | 78 +-
docs/1.0.0/feature/index.xml | 8 +-
docs/1.0.0/feature/observability.html | 275 ++---
docs/1.0.0/feature/recon.html | 22 +
docs/1.0.0/feature/topology.html | 78 +-
docs/1.0.0/index.html | 27 +-
docs/1.0.0/index.xml | 29 +-
docs/1.0.0/interface.html | 24 +-
docs/1.0.0/interface/cli.html | 202 ++--
docs/1.0.0/interface/csi.html | 60 +-
docs/1.0.0/interface/index.xml | 11 +-
docs/1.0.0/interface/javaapi.html | 162 +--
docs/1.0.0/interface/o3fs.html | 64 +-
docs/1.0.0/interface/ofs.html | 70 +-
docs/1.0.0/interface/s3.html | 82 +-
docs/1.0.0/recipe.html | 22 +
docs/1.0.0/recipe/index.xml | 8 +-
docs/1.0.0/recipe/prometheus.html | 66 +-
docs/1.0.0/recipe/sparkozonefsk8s.html | 110 +-
docs/1.0.0/security.html | 22 +
docs/1.0.0/security/index.xml | 8 +-
docs/1.0.0/security/secureozone.html | 30 +-
docs/1.0.0/security/securingdatanodes.html | 22 +
docs/1.0.0/security/securingozonehttp.html | 22 +
docs/1.0.0/security/securings3.html | 36 +-
docs/1.0.0/security/securingtde.html | 30 +-
docs/1.0.0/security/securityacls.html | 22 +
docs/1.0.0/security/securitywithranger.html | 22 +
docs/1.0.0/sitemap.xml | 30 +-
docs/1.0.0/start.html | 22 +
docs/1.0.0/start/fromsource.html | 38 +-
docs/1.0.0/start/index.xml | 26 +-
docs/1.0.0/start/kubernetes.html | 28 +-
docs/1.0.0/start/minikube.html | 50 +-
docs/1.0.0/start/onprem.html | 73 +-
docs/1.0.0/start/runningviadocker.html | 34 +-
docs/1.0.0/start/startfromdockerhub.html | 56 +-
docs/1.0.0/tools.html | 22 +
docs/1.0.0/tools/admin.html | 22 +
docs/1.0.0/tools/auditparser.html | 32 +-
docs/1.0.0/tools/genconf.html | 22 +
docs/1.0.0/tools/index.xml | 8 +-
docs/1.0.0/tools/testtools.html | 36 +-
docs/1.0.0/zh/concept.html | 22 +
docs/1.0.0/zh/concept/datanodes.html | 22 +
docs/1.0.0/zh/concept/index.xml | 8 +-
docs/1.0.0/zh/concept/overview.html | 22 +
docs/1.0.0/zh/concept/ozonemanager.html | 22 +
docs/1.0.0/zh/concept/storagecontainermanager.html | 22 +
docs/1.0.0/zh/feature.html | 22 +
docs/1.0.0/zh/feature/gdpr.html | 22 +
docs/1.0.0/zh/feature/index.xml | 8 +-
docs/1.0.0/zh/index.html | 27 +-
docs/1.0.0/zh/index.xml | 38 +-
docs/1.0.0/zh/interface.html | 22 +
docs/1.0.0/zh/interface/index.xml | 9 +-
docs/1.0.0/zh/interface/javaapi.html | 162 +--
docs/1.0.0/zh/interface/o3fs.html | 92 +-
docs/1.0.0/zh/interface/s3.html | 66 +-
docs/1.0.0/zh/recipe.html | 22 +
docs/1.0.0/zh/recipe/index.xml | 8 +-
docs/1.0.0/zh/recipe/prometheus.html | 66 +-
docs/1.0.0/zh/recipe/sparkozonefsk8s.html | 104 +-
docs/1.0.0/zh/security.html | 22 +
docs/1.0.0/zh/security/index.xml | 8 +-
docs/1.0.0/zh/security/secureozone.html | 30 +-
docs/1.0.0/zh/security/securings3.html | 36 +-
docs/1.0.0/zh/security/securingtde.html | 30 +-
docs/1.0.0/zh/security/securityacls.html | 22 +
docs/1.0.0/zh/security/securitywithranger.html | 22 +
docs/1.0.0/zh/sitemap.xml | 191 ++--
docs/1.0.0/zh/start.html | 22 +
docs/1.0.0/zh/start/fromsource.html | 38 +-
docs/1.0.0/zh/start/index.xml | 37 +-
docs/1.0.0/zh/start/kubernetes.html | 28 +-
docs/1.0.0/zh/start/minikube.html | 50 +-
docs/1.0.0/zh/start/onprem.html | 73 +-
docs/1.0.0/zh/start/runningviadocker.html | 34 +-
docs/1.0.0/zh/start/startfromdockerhub.html | 56 +-
docs/1.0.0/zh/tools.html | 22 +
docs/1.0.0/zh/tools/auditparser.html | 32 +-
docs/1.0.0/zh/tools/genconf.html | 22 +
docs/1.0.0/zh/tools/index.xml | 8 +-
docs/1.0.0/zh/tools/testtools.html | 36 +-
docs/1.1.0/concept.html | 24 +-
docs/1.1.0/concept/containers.html | 24 +-
docs/1.1.0/concept/datanodes.html | 24 +-
docs/1.1.0/concept/overview.html | 24 +-
docs/1.1.0/concept/ozonemanager.html | 24 +-
docs/1.1.0/concept/recon.html | 33 +-
docs/1.1.0/concept/storagecontainermanager.html | 24 +-
docs/1.1.0/design.html | 22 +
docs/1.1.0/design/configless.html | 24 +-
docs/1.1.0/design/decommissioning.html | 123 ++-
docs/1.1.0/design/ec.html | 24 +-
docs/1.1.0/design/gdpr.html | 24 +-
docs/1.1.0/design/locks.html | 24 +-
docs/1.1.0/design/multiraft.html | 24 +-
docs/1.1.0/design/namespace-support.html | 24 +-
docs/1.1.0/design/nfs.html | 24 +-
docs/1.1.0/design/ofs.html | 24 +-
docs/1.1.0/design/omha.html | 24 +-
docs/1.1.0/design/ozone-enhancement-proposals.html | 57 +-
docs/1.1.0/design/recon1.html | 24 +-
docs/1.1.0/design/recon2.html | 24 +-
docs/1.1.0/design/s3-performance.html | 38 +-
docs/1.1.0/design/s3gateway.html | 24 +-
docs/1.1.0/design/scmha.html | 24 +-
docs/1.1.0/design/storage-class.html | 24 +-
docs/1.1.0/design/tde.html | 24 +-
docs/1.1.0/design/token.html | 24 +-
docs/1.1.0/design/topology.html | 24 +-
docs/1.1.0/design/trash.html | 24 +-
docs/1.1.0/design/typesafeconfig.html | 24 +-
docs/1.1.0/design/volume-management.html | 38 +-
docs/1.1.0/en/index.html | 11 +-
docs/1.1.0/feature.html | 24 +-
docs/1.1.0/feature/ha.html | 80 +-
docs/1.1.0/feature/observability.html | 277 ++---
docs/1.1.0/feature/quota.html | 80 +-
docs/1.1.0/feature/recon.html | 26 +-
docs/1.1.0/feature/topology.html | 80 +-
docs/1.1.0/index.html | 29 +-
docs/1.1.0/index.xml | 20 +-
docs/1.1.0/interface.html | 26 +-
docs/1.1.0/interface/cli.html | 198 ++--
docs/1.1.0/interface/csi.html | 62 +-
docs/1.1.0/interface/index.xml | 2 +-
docs/1.1.0/interface/javaapi.html | 164 +--
docs/1.1.0/interface/o3fs.html | 66 +-
docs/1.1.0/interface/ofs.html | 72 +-
docs/1.1.0/interface/reconapi.html | 484 +++++----
docs/1.1.0/interface/s3.html | 82 +-
docs/1.1.0/recipe.html | 24 +-
docs/1.1.0/recipe/botoclient.html | 24 +-
docs/1.1.0/recipe/prometheus.html | 68 +-
docs/1.1.0/recipe/sparkozonefsk8s.html | 112 +-
docs/1.1.0/security.html | 24 +-
docs/1.1.0/security/gdpr.html | 32 +-
docs/1.1.0/security/secureozone.html | 24 +-
docs/1.1.0/security/securingdatanodes.html | 24 +-
docs/1.1.0/security/securingozonehttp.html | 24 +-
docs/1.1.0/security/securings3.html | 38 +-
docs/1.1.0/security/securingtde.html | 32 +-
docs/1.1.0/security/securityacls.html | 24 +-
docs/1.1.0/security/securitywithranger.html | 24 +-
docs/1.1.0/start.html | 24 +-
docs/1.1.0/start/fromsource.html | 40 +-
docs/1.1.0/start/index.xml | 18 +-
docs/1.1.0/start/kubernetes.html | 30 +-
docs/1.1.0/start/minikube.html | 52 +-
docs/1.1.0/start/onprem.html | 75 +-
docs/1.1.0/start/runningviadocker.html | 36 +-
docs/1.1.0/start/startfromdockerhub.html | 58 +-
docs/1.1.0/tools.html | 24 +-
docs/1.1.0/tools/admin.html | 24 +-
docs/1.1.0/tools/auditparser.html | 34 +-
docs/1.1.0/tools/genconf.html | 24 +-
docs/1.1.0/tools/testtools.html | 38 +-
docs/1.1.0/zh/concept.html | 24 +-
docs/1.1.0/zh/concept/containers.html | 24 +-
docs/1.1.0/zh/concept/datanodes.html | 24 +-
docs/1.1.0/zh/concept/overview.html | 24 +-
docs/1.1.0/zh/concept/ozonemanager.html | 24 +-
docs/1.1.0/zh/concept/recon.html | 33 +-
docs/1.1.0/zh/concept/storagecontainermanager.html | 24 +-
docs/1.1.0/zh/feature.html | 24 +-
docs/1.1.0/zh/feature/ha.html | 80 +-
docs/1.1.0/zh/feature/quota.html | 80 +-
docs/1.1.0/zh/feature/recon.html | 26 +-
docs/1.1.0/zh/index.html | 29 +-
docs/1.1.0/zh/index.xml | 29 +-
docs/1.1.0/zh/interface.html | 24 +-
docs/1.1.0/zh/interface/csi.html | 62 +-
docs/1.1.0/zh/interface/javaapi.html | 164 +--
docs/1.1.0/zh/interface/o3fs.html | 92 +-
docs/1.1.0/zh/interface/reconapi.html | 484 +++++----
docs/1.1.0/zh/interface/s3.html | 80 +-
docs/1.1.0/zh/recipe.html | 24 +-
docs/1.1.0/zh/recipe/prometheus.html | 68 +-
docs/1.1.0/zh/recipe/sparkozonefsk8s.html | 106 +-
docs/1.1.0/zh/security.html | 24 +-
docs/1.1.0/zh/security/gdpr.html | 24 +-
docs/1.1.0/zh/security/secureozone.html | 24 +-
docs/1.1.0/zh/security/securingdatanodes.html | 24 +-
docs/1.1.0/zh/security/securings3.html | 38 +-
docs/1.1.0/zh/security/securingtde.html | 32 +-
docs/1.1.0/zh/security/securityacls.html | 24 +-
docs/1.1.0/zh/security/securitywithranger.html | 24 +-
docs/1.1.0/zh/sitemap.xml | 32 +-
docs/1.1.0/zh/start.html | 24 +-
docs/1.1.0/zh/start/fromsource.html | 40 +-
docs/1.1.0/zh/start/index.xml | 29 +-
docs/1.1.0/zh/start/kubernetes.html | 30 +-
docs/1.1.0/zh/start/minikube.html | 52 +-
docs/1.1.0/zh/start/onprem.html | 75 +-
docs/1.1.0/zh/start/runningviadocker.html | 36 +-
docs/1.1.0/zh/start/startfromdockerhub.html | 58 +-
docs/1.1.0/zh/tools.html | 24 +-
docs/1.1.0/zh/tools/auditparser.html | 34 +-
docs/1.1.0/zh/tools/genconf.html | 24 +-
docs/1.1.0/zh/tools/testtools.html | 38 +-
docs/1.2.0/concept.html | 24 +-
docs/1.2.0/concept/containers.html | 24 +-
docs/1.2.0/concept/datanodes.html | 24 +-
docs/1.2.0/concept/overview.html | 24 +-
docs/1.2.0/concept/ozonemanager.html | 24 +-
docs/1.2.0/concept/recon.html | 24 +-
docs/1.2.0/concept/storagecontainermanager.html | 24 +-
docs/1.2.0/design.html | 22 +
docs/1.2.0/design/configless.html | 24 +-
docs/1.2.0/design/decommissioning.html | 95 +-
docs/1.2.0/design/ec.html | 24 +-
docs/1.2.0/design/gdpr.html | 24 +-
docs/1.2.0/design/locks.html | 24 +-
docs/1.2.0/design/multiraft.html | 24 +-
docs/1.2.0/design/namespace-support.html | 24 +-
docs/1.2.0/design/nfs.html | 24 +-
docs/1.2.0/design/nonrolling-upgrade.html | 24 +-
docs/1.2.0/design/ofs.html | 24 +-
docs/1.2.0/design/omha.html | 24 +-
docs/1.2.0/design/omprepare.html | 24 +-
docs/1.2.0/design/ozone-enhancement-proposals.html | 57 +-
docs/1.2.0/design/recon1.html | 24 +-
docs/1.2.0/design/recon2.html | 24 +-
docs/1.2.0/design/s3-performance.html | 24 +-
docs/1.2.0/design/s3gateway.html | 24 +-
docs/1.2.0/design/scmha.html | 24 +-
docs/1.2.0/design/storage-class.html | 24 +-
docs/1.2.0/design/tde.html | 24 +-
docs/1.2.0/design/token.html | 24 +-
docs/1.2.0/design/topology.html | 24 +-
docs/1.2.0/design/trash.html | 24 +-
docs/1.2.0/design/typesafeconfig.html | 24 +-
docs/1.2.0/design/upgrade-dev-primer.html | 24 +-
docs/1.2.0/design/volume-management.html | 26 +-
docs/1.2.0/en/index.html | 11 +-
docs/1.2.0/feature.html | 24 +-
docs/1.2.0/feature/nonrolling-upgrade.html | 24 +-
docs/1.2.0/feature/observability.html | 277 ++---
docs/1.2.0/feature/om-ha.html | 78 +-
docs/1.2.0/feature/prefixfso.html | 42 +-
docs/1.2.0/feature/quota.html | 80 +-
docs/1.2.0/feature/recon.html | 26 +-
docs/1.2.0/feature/scm-ha.html | 140 +--
docs/1.2.0/feature/topology.html | 80 +-
docs/1.2.0/index.html | 26 +-
docs/1.2.0/index.xml | 20 +-
docs/1.2.0/interface.html | 26 +-
docs/1.2.0/interface/cli.html | 198 ++--
docs/1.2.0/interface/csi.html | 62 +-
docs/1.2.0/interface/index.xml | 2 +-
docs/1.2.0/interface/javaapi.html | 164 +--
docs/1.2.0/interface/o3fs.html | 66 +-
docs/1.2.0/interface/ofs.html | 68 +-
docs/1.2.0/interface/reconapi.html | 662 ++++++------
docs/1.2.0/interface/s3.html | 108 +-
docs/1.2.0/recipe.html | 24 +-
docs/1.2.0/recipe/botoclient.html | 24 +-
docs/1.2.0/recipe/prometheus.html | 68 +-
docs/1.2.0/recipe/sparkozonefsk8s.html | 106 +-
docs/1.2.0/security.html | 24 +-
docs/1.2.0/security/gdpr.html | 32 +-
docs/1.2.0/security/secureozone.html | 24 +-
docs/1.2.0/security/securingdatanodes.html | 24 +-
docs/1.2.0/security/securingozonehttp.html | 24 +-
docs/1.2.0/security/securings3.html | 38 +-
docs/1.2.0/security/securingtde.html | 47 +-
docs/1.2.0/security/securityacls.html | 24 +-
docs/1.2.0/security/securitywithranger.html | 24 +-
docs/1.2.0/start.html | 24 +-
docs/1.2.0/start/fromsource.html | 40 +-
docs/1.2.0/start/index.xml | 18 +-
docs/1.2.0/start/kubernetes.html | 24 +-
docs/1.2.0/start/minikube.html | 46 +-
docs/1.2.0/start/onprem.html | 75 +-
docs/1.2.0/start/runningviadocker.html | 36 +-
docs/1.2.0/start/startfromdockerhub.html | 58 +-
docs/1.2.0/tools.html | 24 +-
docs/1.2.0/tools/admin.html | 24 +-
docs/1.2.0/tools/auditparser.html | 34 +-
docs/1.2.0/tools/genconf.html | 24 +-
docs/1.2.0/tools/testtools.html | 28 +-
docs/1.2.0/zh/concept.html | 24 +-
docs/1.2.0/zh/concept/containers.html | 24 +-
docs/1.2.0/zh/concept/datanodes.html | 24 +-
docs/1.2.0/zh/concept/overview.html | 24 +-
docs/1.2.0/zh/concept/ozonemanager.html | 24 +-
docs/1.2.0/zh/concept/recon.html | 24 +-
docs/1.2.0/zh/concept/storagecontainermanager.html | 24 +-
docs/1.2.0/zh/feature.html | 24 +-
docs/1.2.0/zh/feature/om-ha.html | 78 +-
docs/1.2.0/zh/feature/quota.html | 80 +-
docs/1.2.0/zh/feature/recon.html | 26 +-
docs/1.2.0/zh/feature/scm-ha.html | 116 +-
docs/1.2.0/zh/index.html | 26 +-
docs/1.2.0/zh/index.xml | 29 +-
docs/1.2.0/zh/interface.html | 24 +-
docs/1.2.0/zh/interface/csi.html | 62 +-
docs/1.2.0/zh/interface/javaapi.html | 164 +--
docs/1.2.0/zh/interface/o3fs.html | 92 +-
docs/1.2.0/zh/interface/reconapi.html | 460 ++++----
docs/1.2.0/zh/interface/s3.html | 80 +-
docs/1.2.0/zh/recipe.html | 24 +-
docs/1.2.0/zh/recipe/prometheus.html | 68 +-
docs/1.2.0/zh/recipe/sparkozonefsk8s.html | 100 +-
docs/1.2.0/zh/security.html | 24 +-
docs/1.2.0/zh/security/gdpr.html | 24 +-
docs/1.2.0/zh/security/secureozone.html | 24 +-
docs/1.2.0/zh/security/securingdatanodes.html | 24 +-
docs/1.2.0/zh/security/securings3.html | 38 +-
docs/1.2.0/zh/security/securingtde.html | 32 +-
docs/1.2.0/zh/security/securityacls.html | 24 +-
docs/1.2.0/zh/security/securitywithranger.html | 24 +-
docs/1.2.0/zh/sitemap.xml | 38 +-
docs/1.2.0/zh/start.html | 24 +-
docs/1.2.0/zh/start/fromsource.html | 40 +-
docs/1.2.0/zh/start/index.xml | 29 +-
docs/1.2.0/zh/start/kubernetes.html | 24 +-
docs/1.2.0/zh/start/minikube.html | 46 +-
docs/1.2.0/zh/start/onprem.html | 75 +-
docs/1.2.0/zh/start/runningviadocker.html | 36 +-
docs/1.2.0/zh/start/startfromdockerhub.html | 58 +-
docs/1.2.0/zh/tools.html | 24 +-
docs/1.2.0/zh/tools/auditparser.html | 34 +-
docs/1.2.0/zh/tools/genconf.html | 24 +-
docs/1.2.0/zh/tools/testtools.html | 28 +-
docs/1.2.1/concept.html | 24 +-
docs/1.2.1/concept/containers.html | 24 +-
docs/1.2.1/concept/datanodes.html | 24 +-
docs/1.2.1/concept/overview.html | 24 +-
docs/1.2.1/concept/ozonemanager.html | 24 +-
docs/1.2.1/concept/recon.html | 24 +-
docs/1.2.1/concept/storagecontainermanager.html | 24 +-
docs/1.2.1/design.html | 22 +
docs/1.2.1/design/configless.html | 24 +-
docs/1.2.1/design/decommissioning.html | 95 +-
docs/1.2.1/design/ec.html | 24 +-
docs/1.2.1/design/gdpr.html | 24 +-
docs/1.2.1/design/locks.html | 24 +-
docs/1.2.1/design/multiraft.html | 24 +-
docs/1.2.1/design/namespace-support.html | 24 +-
docs/1.2.1/design/nfs.html | 24 +-
docs/1.2.1/design/nonrolling-upgrade.html | 24 +-
docs/1.2.1/design/ofs.html | 24 +-
docs/1.2.1/design/omha.html | 24 +-
docs/1.2.1/design/omprepare.html | 24 +-
docs/1.2.1/design/ozone-enhancement-proposals.html | 57 +-
docs/1.2.1/design/recon1.html | 24 +-
docs/1.2.1/design/recon2.html | 24 +-
docs/1.2.1/design/s3-performance.html | 24 +-
docs/1.2.1/design/s3gateway.html | 24 +-
docs/1.2.1/design/scmha.html | 24 +-
docs/1.2.1/design/storage-class.html | 24 +-
docs/1.2.1/design/tde.html | 24 +-
docs/1.2.1/design/token.html | 24 +-
docs/1.2.1/design/topology.html | 24 +-
docs/1.2.1/design/trash.html | 24 +-
docs/1.2.1/design/typesafeconfig.html | 24 +-
docs/1.2.1/design/upgrade-dev-primer.html | 24 +-
docs/1.2.1/design/volume-management.html | 27 +-
docs/1.2.1/en/index.html | 11 +-
docs/1.2.1/feature.html | 24 +-
docs/1.2.1/feature/nonrolling-upgrade.html | 24 +-
docs/1.2.1/feature/observability.html | 277 ++---
docs/1.2.1/feature/om-ha.html | 78 +-
docs/1.2.1/feature/prefixfso.html | 42 +-
docs/1.2.1/feature/quota.html | 80 +-
docs/1.2.1/feature/recon.html | 26 +-
docs/1.2.1/feature/scm-ha.html | 140 +--
docs/1.2.1/feature/topology.html | 80 +-
docs/1.2.1/index.html | 26 +-
docs/1.2.1/index.xml | 20 +-
docs/1.2.1/interface.html | 26 +-
docs/1.2.1/interface/cli.html | 198 ++--
docs/1.2.1/interface/csi.html | 62 +-
docs/1.2.1/interface/index.xml | 2 +-
docs/1.2.1/interface/javaapi.html | 164 +--
docs/1.2.1/interface/o3fs.html | 66 +-
docs/1.2.1/interface/ofs.html | 68 +-
docs/1.2.1/interface/reconapi.html | 662 ++++++------
docs/1.2.1/interface/s3.html | 108 +-
docs/1.2.1/recipe.html | 24 +-
docs/1.2.1/recipe/botoclient.html | 24 +-
docs/1.2.1/recipe/prometheus.html | 68 +-
docs/1.2.1/recipe/sparkozonefsk8s.html | 104 +-
docs/1.2.1/security.html | 24 +-
docs/1.2.1/security/gdpr.html | 32 +-
docs/1.2.1/security/secureozone.html | 24 +-
docs/1.2.1/security/securingdatanodes.html | 24 +-
docs/1.2.1/security/securingozonehttp.html | 24 +-
docs/1.2.1/security/securings3.html | 38 +-
docs/1.2.1/security/securingtde.html | 47 +-
docs/1.2.1/security/securityacls.html | 24 +-
docs/1.2.1/security/securitywithranger.html | 24 +-
docs/1.2.1/start.html | 24 +-
docs/1.2.1/start/fromsource.html | 40 +-
docs/1.2.1/start/index.xml | 18 +-
docs/1.2.1/start/kubernetes.html | 24 +-
docs/1.2.1/start/minikube.html | 46 +-
docs/1.2.1/start/onprem.html | 75 +-
docs/1.2.1/start/runningviadocker.html | 36 +-
docs/1.2.1/start/startfromdockerhub.html | 58 +-
docs/1.2.1/tools.html | 24 +-
docs/1.2.1/tools/admin.html | 24 +-
docs/1.2.1/tools/auditparser.html | 34 +-
docs/1.2.1/tools/genconf.html | 24 +-
docs/1.2.1/tools/testtools.html | 28 +-
docs/1.2.1/zh/concept.html | 24 +-
docs/1.2.1/zh/concept/containers.html | 24 +-
docs/1.2.1/zh/concept/datanodes.html | 24 +-
docs/1.2.1/zh/concept/overview.html | 24 +-
docs/1.2.1/zh/concept/ozonemanager.html | 24 +-
docs/1.2.1/zh/concept/recon.html | 24 +-
docs/1.2.1/zh/concept/storagecontainermanager.html | 24 +-
docs/1.2.1/zh/feature.html | 24 +-
docs/1.2.1/zh/feature/om-ha.html | 78 +-
docs/1.2.1/zh/feature/quota.html | 80 +-
docs/1.2.1/zh/feature/recon.html | 26 +-
docs/1.2.1/zh/feature/scm-ha.html | 116 +-
docs/1.2.1/zh/index.html | 26 +-
docs/1.2.1/zh/index.xml | 29 +-
docs/1.2.1/zh/interface.html | 24 +-
docs/1.2.1/zh/interface/csi.html | 62 +-
docs/1.2.1/zh/interface/javaapi.html | 164 +--
docs/1.2.1/zh/interface/o3fs.html | 90 +-
docs/1.2.1/zh/interface/reconapi.html | 460 ++++----
docs/1.2.1/zh/interface/s3.html | 80 +-
docs/1.2.1/zh/recipe.html | 24 +-
docs/1.2.1/zh/recipe/prometheus.html | 68 +-
docs/1.2.1/zh/recipe/sparkozonefsk8s.html | 100 +-
docs/1.2.1/zh/security.html | 24 +-
docs/1.2.1/zh/security/gdpr.html | 24 +-
docs/1.2.1/zh/security/secureozone.html | 24 +-
docs/1.2.1/zh/security/securingdatanodes.html | 24 +-
docs/1.2.1/zh/security/securings3.html | 38 +-
docs/1.2.1/zh/security/securingtde.html | 32 +-
docs/1.2.1/zh/security/securityacls.html | 24 +-
docs/1.2.1/zh/security/securitywithranger.html | 24 +-
docs/1.2.1/zh/sitemap.xml | 38 +-
docs/1.2.1/zh/start.html | 24 +-
docs/1.2.1/zh/start/fromsource.html | 40 +-
docs/1.2.1/zh/start/index.xml | 29 +-
docs/1.2.1/zh/start/kubernetes.html | 24 +-
docs/1.2.1/zh/start/minikube.html | 46 +-
docs/1.2.1/zh/start/onprem.html | 75 +-
docs/1.2.1/zh/start/runningviadocker.html | 36 +-
docs/1.2.1/zh/start/startfromdockerhub.html | 58 +-
docs/1.2.1/zh/tools.html | 24 +-
docs/1.2.1/zh/tools/auditparser.html | 34 +-
docs/1.2.1/zh/tools/genconf.html | 24 +-
docs/1.2.1/zh/tools/testtools.html | 28 +-
docs/1.3.0/concept.html | 46 +-
docs/1.3.0/concept/containers.html | 24 +-
docs/1.3.0/concept/datanodes.html | 24 +-
docs/1.3.0/concept/overview.html | 24 +-
docs/1.3.0/concept/ozonemanager.html | 24 +-
docs/1.3.0/concept/recon.html | 24 +-
docs/1.3.0/concept/storagecontainermanager.html | 24 +-
docs/1.3.0/design.html | 44 +-
docs/1.3.0/design/configless.html | 24 +-
docs/1.3.0/design/decommissioning.html | 123 ++-
docs/1.3.0/design/dn-merge-rocksdb.html | 24 +-
docs/1.3.0/design/ec.html | 24 +-
docs/1.3.0/design/gdpr.html | 24 +-
docs/1.3.0/design/locks.html | 24 +-
docs/1.3.0/design/multiraft.html | 24 +-
docs/1.3.0/design/namespace-support.html | 24 +-
docs/1.3.0/design/nfs.html | 24 +-
docs/1.3.0/design/nonrolling-upgrade.html | 24 +-
docs/1.3.0/design/ofs.html | 24 +-
docs/1.3.0/design/omha.html | 24 +-
docs/1.3.0/design/omprepare.html | 24 +-
docs/1.3.0/design/ozone-enhancement-proposals.html | 57 +-
docs/1.3.0/design/recon1.html | 24 +-
docs/1.3.0/design/recon2.html | 24 +-
docs/1.3.0/design/s3-performance.html | 38 +-
docs/1.3.0/design/s3gateway.html | 24 +-
docs/1.3.0/design/scmha.html | 24 +-
docs/1.3.0/design/storage-class.html | 24 +-
docs/1.3.0/design/tde.html | 24 +-
docs/1.3.0/design/token.html | 24 +-
docs/1.3.0/design/topology.html | 24 +-
docs/1.3.0/design/trash.html | 24 +-
docs/1.3.0/design/typesafeconfig.html | 24 +-
docs/1.3.0/design/upgrade-dev-primer.html | 24 +-
docs/1.3.0/design/volume-management.html | 38 +-
docs/1.3.0/en/index.html | 11 +-
docs/1.3.0/feature.html | 46 +-
docs/1.3.0/feature/dn-merge-rocksdb.html | 60 +-
docs/1.3.0/feature/erasurecoding.html | 56 +-
docs/1.3.0/feature/nonrolling-upgrade.html | 36 +-
docs/1.3.0/feature/observability.html | 277 ++---
docs/1.3.0/feature/om-ha.html | 100 +-
docs/1.3.0/feature/prefixfso.html | 40 +-
docs/1.3.0/feature/quota.html | 80 +-
docs/1.3.0/feature/recon.html | 26 +-
.../feature/s3-multi-tenancy-access-control.html | 26 +-
docs/1.3.0/feature/s3-multi-tenancy-setup.html | 82 +-
docs/1.3.0/feature/s3-multi-tenancy.html | 24 +-
docs/1.3.0/feature/s3-tenant-commands.html | 430 ++++----
docs/1.3.0/feature/scm-ha.html | 144 +--
docs/1.3.0/feature/topology.html | 80 +-
docs/1.3.0/index.html | 46 +-
docs/1.3.0/index.xml | 20 +-
docs/1.3.0/interface.html | 48 +-
docs/1.3.0/interface/cli.html | 200 ++--
docs/1.3.0/interface/csi.html | 62 +-
docs/1.3.0/interface/index.xml | 2 +-
docs/1.3.0/interface/javaapi.html | 164 +--
docs/1.3.0/interface/o3fs.html | 66 +-
docs/1.3.0/interface/ofs.html | 88 +-
docs/1.3.0/interface/reconapi.html | 662 ++++++------
docs/1.3.0/interface/s3.html | 106 +-
docs/1.3.0/recipe.html | 46 +-
docs/1.3.0/recipe/botoclient.html | 24 +-
docs/1.3.0/recipe/prometheus.html | 68 +-
docs/1.3.0/recipe/sparkozonefsk8s.html | 112 +-
docs/1.3.0/security.html | 46 +-
docs/1.3.0/security/gdpr.html | 32 +-
docs/1.3.0/security/secureozone.html | 24 +-
docs/1.3.0/security/securingdatanodes.html | 24 +-
docs/1.3.0/security/securingozonehttp.html | 24 +-
docs/1.3.0/security/securings3.html | 38 +-
docs/1.3.0/security/securingtde.html | 60 +-
docs/1.3.0/security/securityacls.html | 24 +-
docs/1.3.0/security/securitywithranger.html | 26 +-
docs/1.3.0/start.html | 46 +-
docs/1.3.0/start/fromsource.html | 40 +-
docs/1.3.0/start/index.xml | 18 +-
docs/1.3.0/start/kubernetes.html | 30 +-
docs/1.3.0/start/minikube.html | 52 +-
docs/1.3.0/start/onprem.html | 75 +-
docs/1.3.0/start/runningviadocker.html | 36 +-
docs/1.3.0/start/startfromdockerhub.html | 58 +-
docs/1.3.0/tools.html | 46 +-
docs/1.3.0/tools/admin.html | 24 +-
docs/1.3.0/tools/auditparser.html | 34 +-
docs/1.3.0/tools/genconf.html | 24 +-
docs/1.3.0/tools/testtools.html | 32 +-
docs/1.3.0/zh/concept.html | 24 +-
docs/1.3.0/zh/concept/containers.html | 24 +-
docs/1.3.0/zh/concept/datanodes.html | 24 +-
docs/1.3.0/zh/concept/overview.html | 24 +-
docs/1.3.0/zh/concept/ozonemanager.html | 24 +-
docs/1.3.0/zh/concept/recon.html | 24 +-
docs/1.3.0/zh/concept/storagecontainermanager.html | 24 +-
docs/1.3.0/zh/feature.html | 24 +-
docs/1.3.0/zh/feature/observability.html | 277 ++---
docs/1.3.0/zh/feature/om-ha.html | 88 +-
docs/1.3.0/zh/feature/prefixfso.html | 40 +-
docs/1.3.0/zh/feature/quota.html | 80 +-
docs/1.3.0/zh/feature/recon.html | 26 +-
docs/1.3.0/zh/feature/scm-ha.html | 144 +--
docs/1.3.0/zh/index.html | 26 +-
docs/1.3.0/zh/index.xml | 45 +-
docs/1.3.0/zh/interface.html | 24 +-
docs/1.3.0/zh/interface/csi.html | 62 +-
docs/1.3.0/zh/interface/javaapi.html | 164 +--
docs/1.3.0/zh/interface/o3fs.html | 92 +-
docs/1.3.0/zh/interface/reconapi.html | 460 ++++----
docs/1.3.0/zh/interface/s3.html | 80 +-
docs/1.3.0/zh/recipe.html | 24 +-
docs/1.3.0/zh/recipe/prometheus.html | 68 +-
docs/1.3.0/zh/recipe/sparkozonefsk8s.html | 106 +-
docs/1.3.0/zh/security.html | 24 +-
docs/1.3.0/zh/security/gdpr.html | 24 +-
docs/1.3.0/zh/security/secureozone.html | 24 +-
docs/1.3.0/zh/security/securingdatanodes.html | 24 +-
docs/1.3.0/zh/security/securings3.html | 38 +-
docs/1.3.0/zh/security/securingtde.html | 32 +-
docs/1.3.0/zh/security/securityacls.html | 24 +-
docs/1.3.0/zh/security/securitywithranger.html | 26 +-
docs/1.3.0/zh/sitemap.xml | 56 +-
docs/1.3.0/zh/start.html | 24 +-
docs/1.3.0/zh/start/fromsource.html | 40 +-
docs/1.3.0/zh/start/index.xml | 29 +-
docs/1.3.0/zh/start/kubernetes.html | 30 +-
docs/1.3.0/zh/start/minikube.html | 52 +-
docs/1.3.0/zh/start/onprem.html | 75 +-
docs/1.3.0/zh/start/runningviadocker.html | 36 +-
docs/1.3.0/zh/start/startfromdockerhub.html | 58 +-
docs/1.3.0/zh/tools.html | 24 +-
docs/1.3.0/zh/tools/auditparser.html | 34 +-
docs/1.3.0/zh/tools/genconf.html | 24 +-
docs/1.3.0/zh/tools/testtools.html | 32 +-
712 files changed, 25227 insertions(+), 12531 deletions(-)
diff --git a/docs/0.5.0-beta/beyond.html b/docs/0.5.0-beta/beyond.html
index 3d7b56f..62a9422 100644
--- a/docs/0.5.0-beta/beyond.html
+++ b/docs/0.5.0-beta/beyond.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -221,7 +243,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">Beyond Basics </h3>
diff --git a/docs/0.5.0-beta/beyond/containers.html b/docs/0.5.0-beta/beyond/containers.html
index f32e700..1dff569 100644
--- a/docs/0.5.0-beta/beyond/containers.html
+++ b/docs/0.5.0-beta/beyond/containers.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Ozone Containers</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,164 +251,122 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Docker heavily is used at the ozone development with three principal use-cases:</p>
-
<ul>
<li><strong>dev</strong>:
-
<ul>
<li>We use docker to start local pseudo-clusters (docker provides unified environment, but no image creation is required)</li>
-</ul></li>
+</ul>
+</li>
<li><strong>test</strong>:
-
<ul>
<li>We create docker images from the dev branches to test ozone in kubernetes and other container orchestrator system</li>
<li>We provide <em>apache/ozone</em> images for each release to make it easier for evaluation of Ozone.
These images are <strong>not</strong> created <strong>for production</strong> usage.</li>
-</ul></li>
</ul>
-
+</li>
+</ul>
<div class="alert alert-warning" role="alert">
We <b>strongly</b> recommend that you create your own custom images when you
deploy ozone into production using containers. Please treat all the standard
shipped container images and k8s resources as examples and guides to help you
customize your own deployment.
</div>
-
<ul>
<li><strong>production</strong>:
-
<ul>
<li>We have documentation on how you can create your own docker image for your production cluster.</li>
-</ul></li>
</ul>
-
+</li>
+</ul>
<p>Let’s check out each of the use-cases in more detail:</p>
-
<h2 id="development">Development</h2>
-
<p>Ozone artifact contains example docker-compose directories to make it easier to start Ozone cluster in your local machine.</p>
-
<p>From distribution:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd compose/ozone
-docker-compose up -d</code></pre></div>
-<p>After a local build:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd hadoop-ozone/dist/target/ozone-*/compose
-docker-compose up -d</code></pre></div>
-<p>These environments are very important tools to start different type of Ozone clusters at any time.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd compose/ozone
+</span></span><span style="display:flex;"><span>docker-compose up -d
+</span></span></code></pre></div><p>After a local build:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd hadoop-ozone/dist/target/ozone-*/compose
+</span></span><span style="display:flex;"><span>docker-compose up -d
+</span></span></code></pre></div><p>These environments are very important tools to start different type of Ozone clusters at any time.</p>
<p>To be sure that the compose files are up-to-date, we also provide acceptance test suites which start
the cluster and check the basic behaviour.</p>
-
<p>The acceptance tests are part of the distribution, and you can find the test definitions in <code>smoketest</code> directory.</p>
-
<p>You can start the tests from any compose directory:</p>
-
<p>For example:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd compose/ozone
-./test.sh</code></pre></div>
-<h3 id="implementation-details">Implementation details</h3>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd compose/ozone
+</span></span><span style="display:flex;"><span>./test.sh
+</span></span></code></pre></div><h3 id="implementation-details">Implementation details</h3>
<p><code>compose</code> tests are based on the apache/hadoop-runner docker image. The image itself does not contain
any Ozone jar file or binary just the helper scripts to start ozone.</p>
-
<p>hadoop-runner provdes a fixed environment to run Ozone everywhere, but the ozone distribution itself
is mounted from the including directory:</p>
-
<p>(Example docker-compose fragment)</p>
-
-<pre><code> scm:
+<pre tabindex="0"><code> scm:
image: apache/hadoop-runner:jdk11
volumes:
- ../..:/opt/hadoop
ports:
- 9876:9876
-
-</code></pre>
-
-<p>The containers are configured based on environment variables, but because the same environment
+</code></pre><p>The containers are configured based on environment variables, but because the same environment
variables should be set for each containers we maintain the list of the environment variables
in a separated file:</p>
-
-<pre><code> scm:
+<pre tabindex="0"><code> scm:
image: apache/hadoop-runner:jdk11
#...
env_file:
- ./docker-config
-</code></pre>
-
-<p>The docker-config file contains the list of the required environment variables:</p>
-
-<pre><code>OZONE-SITE.XML_ozone.om.address=om
+</code></pre><p>The docker-config file contains the list of the required environment variables:</p>
+<pre tabindex="0"><code>OZONE-SITE.XML_ozone.om.address=om
OZONE-SITE.XML_ozone.om.http-address=om:9874
OZONE-SITE.XML_ozone.scm.names=scm
#...
-</code></pre>
-
-<p>As you can see we use naming convention. Based on the name of the environment variable, the
+</code></pre><p>As you can see we use naming convention. Based on the name of the environment variable, the
appropriate hadoop config XML (<code>ozone-site.xml</code> in our case) will be generated by a
<a href="https://github.com/apache/hadoop/tree/docker-hadoop-runner-latest/scripts">script</a> which is
included in the <code>hadoop-runner</code> base image.</p>
-
<p>The <a href="https://github.com/apache/hadoop/blob/docker-hadoop-runner-latest/scripts/starter.sh">entrypoint</a>
of the <code>hadoop-runner</code> image contains a helper shell script which triggers this transformation and
can do additional actions (eg. initialize scm/om storage, download required keytabs, etc.)
based on environment variables.</p>
-
-<h2 id="test-staging">Test/Staging</h2>
-
+<h2 id="teststaging">Test/Staging</h2>
<p>The <code>docker-compose</code> based approach is recommended only for local test, not for multi node cluster.
To use containers on a multi-node cluster we need a Container Orchestrator like Kubernetes.</p>
-
<p>Kubernetes example files are included in the <code>kubernetes</code> folder.</p>
-
<p><em>Please note</em>: all the provided images are based the <code>hadoop-runner</code> image which contains all the
required tool for testing in staging environments. For production we recommend to create your own,
hardened image with your own base image.</p>
-
<h3 id="test-the-release">Test the release</h3>
-
<p>The release can be tested with deploying any of the example clusters:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd kubernetes/examples/ozone
-kubectl apply -f</code></pre></div>
-<p>Plese note that in this case the latest released container will be downloaded from the dockerhub.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd kubernetes/examples/ozone
+</span></span><span style="display:flex;"><span>kubectl apply -f
+</span></span></code></pre></div><p>Plese note that in this case the latest released container will be downloaded from the dockerhub.</p>
<h3 id="test-the-development-build">Test the development build</h3>
-
<p>To test a development build you can create your own image and upload it to your own docker registry:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mvn clean install -DskipTests -Pdocker-build,docker-push -Ddocker.image<span style="color:#f92672">=</span>myregistry:9000/name/ozone</code></pre></div>
-<p>The configured image will be used in all the generated kubernetes resources files (<code>image:</code> keys are adjusted during the build)</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd kubernetes/examples/ozone
-kubectl apply -f</code></pre></div>
-<h2 id="production">Production</h2>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>mvn clean install -DskipTests -Pdocker-build,docker-push -Ddocker.image<span style="color:#f92672">=</span>myregistry:9000/name/ozone
+</span></span></code></pre></div><p>The configured image will be used in all the generated kubernetes resources files (<code>image:</code> keys are adjusted during the build)</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd kubernetes/examples/ozone
+</span></span><span style="display:flex;"><span>kubectl apply -f
+</span></span></code></pre></div><h2 id="production">Production</h2>
<div class="alert alert-danger" role="alert">
We <b>strongly</b> recommend to use your own image in your production cluster
and
adjust base image, umask, security settings, user settings according to your own requirements.
</div>
-
<p>You can use the source of our development images as an example:</p>
-
<ul>
<li><a href="https://github.com/apache/hadoop/blob/docker-hadoop-runner-jdk11/Dockerfile">Base image</a></li>
<li><a href="https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/dist/src/main/docker/Dockerfile">Docker image</a></li>
</ul>
-
<p>Most of the elements are optional and just helper function but to use the provided example
- kubernetes resources you may need the scripts from
- <a href="https://github.com/apache/hadoop/tree/docker-hadoop-runner-jdk11/scripts">here</a></p>
-
+kubernetes resources you may need the scripts from
+<a href="https://github.com/apache/hadoop/tree/docker-hadoop-runner-jdk11/scripts">here</a></p>
<ul>
<li>The two python scripts convert environment variables to real hadoop XML config files</li>
<li>The start.sh executes the python scripts (and other initialization) based on environment variables.</li>
</ul>
-
<h2 id="containers">Containers</h2>
-
<p>Ozone related container images and source locations:</p>
-
<table class="table table-dark">
<thead>
<tr>
diff --git a/docs/0.5.0-beta/beyond/dockercheatsheet.html b/docs/0.5.0-beta/beyond/dockercheatsheet.html
index 049afe4..7d751d2 100644
--- a/docs/0.5.0-beta/beyond/dockercheatsheet.html
+++ b/docs/0.5.0-beta/beyond/dockercheatsheet.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Docker Cheat Sheet</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,53 +251,36 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>In the <code>compose</code> directory of the ozone distribution there are multiple pseudo-cluster setup which
can be used to run Ozone in different way (for example: secure cluster, with tracing enabled,
with prometheus etc.).</p>
-
<p>If the usage is not document in a specific directory the default usage is the following:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd compose/ozone
-docker-compose up -d</code></pre></div>
-<p>The data of the container is ephemeral and deleted together with the docker volumes.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose down</code></pre></div>
-<h2 id="useful-docker-ozone-commands">Useful Docker & Ozone Commands</h2>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd compose/ozone
+</span></span><span style="display:flex;"><span>docker-compose up -d
+</span></span></code></pre></div><p>The data of the container is ephemeral and deleted together with the docker volumes.</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose down
+</span></span></code></pre></div><h2 id="useful-docker--ozone-commands">Useful Docker & Ozone Commands</h2>
<p>If you make any modifications to ozone, the simplest way to test it is to run freon and unit tests.</p>
-
<p>Here are the instructions to run freon in a docker-based cluster.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose exec datanode bash</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose exec datanode bash</span></span></code></pre></div>
<p>This will open a bash shell on the data node container.
Now we can execute freon for load generation.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone freon randomkeys --numOfVolumes<span style="color:#f92672">=</span><span style="color:#ae81ff">10</span> --numOfBuckets <span style="color:#ae81ff">10</span> --numOfKeys <span style="color:#ae81ff">10</span></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone freon randomkeys --numOfVolumes<span style="color:#f92672">=</span><span style="color:#ae81ff">10</span> --numOfBuckets <span style="color:#ae81ff">10</span> --numOfKeys <span style="color:#ae81ff">10</span></span></span></code></pre></div>
<p>Here is a set of helpful commands for working with docker for ozone.
To check the status of the components:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose ps</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose ps</span></span></code></pre></div>
<p>To get logs from a specific node/service:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose logs scm</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose logs scm</span></span></code></pre></div>
<p>As the WebUI ports are forwarded to the external machine, you can check the web UI:</p>
-
<ul>
-<li>For the Storage Container Manager: <a href="http://localhost:9876">http://localhost:9876</a></li>
-<li>For the Ozone Manager: <a href="http://localhost:9874">http://localhost:9874</a></li>
+<li>For the Storage Container Manager: http://localhost:9876</li>
+<li>For the Ozone Manager: http://localhost:9874</li>
<li>For the Datanode: check the port with <code>docker ps</code> (as there could be multiple data nodes, ports are mapped to the ephemeral port range)</li>
</ul>
-
<p>You can start multiple data nodes with:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose scale datanode<span style="color:#f92672">=</span><span style="color:#ae81ff">3</span></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose scale datanode<span style="color:#f92672">=</span><span style="color:#ae81ff">3</span></span></span></code></pre></div>
<p>You can test the commands from the <a href="../shell.html">Ozone CLI</a> after opening a new bash shell in one of the containers:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose exec datanode bash</code></pre></div>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose exec datanode bash</span></span></code></pre></div>
diff --git a/docs/0.5.0-beta/beyond/index.xml b/docs/0.5.0-beta/beyond/index.xml
index 606c950..289ab82 100644
--- a/docs/0.5.0-beta/beyond/index.xml
+++ b/docs/0.5.0-beta/beyond/index.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Beyond Basics on Ozone</title>
@@ -6,11 +6,7 @@
<description>Recent content in Beyond Basics on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
- <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/beyond/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate><atom:link href="/beyond/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Running concurrently with HDFS</title>
<link>/beyond/runningwithhdfs.html</link>
@@ -39,4 +35,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/beyond/runningwithhdfs.html b/docs/0.5.0-beta/beyond/runningwithhdfs.html
index d07406f..0fbeaf7 100644
--- a/docs/0.5.0-beta/beyond/runningwithhdfs.html
+++ b/docs/0.5.0-beta/beyond/runningwithhdfs.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -229,32 +251,23 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone is designed to work with HDFS. So it is easy to deploy ozone in an
existing HDFS cluster.</p>
-
<p>The container manager part of Ozone can run inside DataNodes as a pluggable module
or as a standalone component. This document describe how can it be started as
a HDFS datanode plugin.</p>
-
<p>To activate ozone you should define the service plugin implementation class.</p>
-
<div class="alert alert-warning" role="alert">
<b>Important</b>: It should be added to the <b>hdfs-site.xml</b> as the plugin should
be activated as part of the normal HDFS Datanode bootstrap.
</div>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>dfs.datanode.plugins<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>org.apache.hadoop.ozone.HddsDatanodeService<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>dfs.datanode.plugins<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>org.apache.hadoop.ozone.HddsDatanodeService<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span></span></span></code></pre></div>
<p>You also need to add the jar file under path /opt/ozone/share/ozone/lib/ to the classpath:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">export HADOOP_CLASSPATH<span style="color:#f92672">=</span>/opt/ozone/share/ozone/lib/*.jar</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>export HADOOP_CLASSPATH<span style="color:#f92672">=</span>/opt/ozone/share/ozone/lib/*.jar</span></span></code></pre></div>
<p>To start ozone with HDFS you should start the the following components:</p>
-
<ol>
<li>HDFS Namenode (from Hadoop distribution)</li>
<li>HDFS Datanode (from the Hadoop distribution with the plugin on the
@@ -262,14 +275,10 @@ classpath from the Ozone distribution)</li>
<li>Ozone Manager (from the Ozone distribution)</li>
<li>Storage Container Manager (from the Ozone distribution)</li>
</ol>
-
<p>Please check the log of the datanode whether the HDDS/Ozone plugin is started or
not. Log of datanode should contain something like this:</p>
-
-<pre><code>2018-09-17 16:19:24 INFO HddsDatanodeService:158 - Started plug-in org.apache.hadoop.ozone.web.OzoneHddsDatanodeService@6f94fb9d
-</code></pre>
-
-<div class="alert alert-warning" role="alert">
+<pre tabindex="0"><code>2018-09-17 16:19:24 INFO HddsDatanodeService:158 - Started plug-in org.apache.hadoop.ozone.web.OzoneHddsDatanodeService@6f94fb9d
+</code></pre><div class="alert alert-warning" role="alert">
<b>Note:</b> The current version of Ozone is tested with Hadoop 3.1.
</div>
diff --git a/docs/0.5.0-beta/concept.html b/docs/0.5.0-beta/concept.html
index 47e6e1e..f64a373 100644
--- a/docs/0.5.0-beta/concept.html
+++ b/docs/0.5.0-beta/concept.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -225,7 +247,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">Ozone Architecture </h3>
diff --git a/docs/0.5.0-beta/concept/datanodes.html b/docs/0.5.0-beta/concept/datanodes.html
index 1296b28..3a5c87f 100644
--- a/docs/0.5.0-beta/concept/datanodes.html
+++ b/docs/0.5.0-beta/concept/datanodes.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Datanodes</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,16 +255,12 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Datanodes are the worker bees of Ozone. All data is stored on data nodes.
Clients write data in terms of blocks. Datanode aggregates these blocks into
a storage container. A storage container is the data streams and metadata
about the blocks written by the clients.</p>
-
<h2 id="storage-containers">Storage Containers</h2>
-
-<p><img src="ContainerMetadata.png" alt="FunctionalOzone" /></p>
-
+<p><img src="ContainerMetadata.png" alt="FunctionalOzone"></p>
<p>A storage container is a self-contained super block. It has a list of Ozone
blocks that reside inside it, as well as on-disk files which contain the
actual data streams. This is the default Storage container format. From
@@ -252,38 +268,29 @@ Ozone’s perspective, container is a protocol spec, actual storage layouts
does not matter. In other words, it is trivial to extend or bring new
container layouts. Hence this should be treated as a reference implementation
of containers under Ozone.</p>
-
<h2 id="understanding-ozone-blocks-and-containers">Understanding Ozone Blocks and Containers</h2>
-
<p>When a client wants to read a key from Ozone, the client sends the name of
the key to the Ozone Manager. Ozone manager returns the list of Ozone blocks
that make up that key.</p>
-
<p>An Ozone block contains the container ID and a local ID. The figure below
shows the logical layout out of Ozone block.</p>
-
-<p><img src="OzoneBlock.png" alt="OzoneBlock" /></p>
-
+<p><img src="OzoneBlock.png" alt="OzoneBlock"></p>
<p>The container ID lets the clients discover the location of the container. The
authoritative information about where a container is located is with the
Storage Container Manager (SCM). In most cases, the container location will be
cached by Ozone Manager and will be returned along with the Ozone blocks.</p>
-
<p>Once the client is able to locate the contianer, that is, understand which
data nodes contain this container, the client will connect to the datanode
and read the data stream specified by <em>Container ID:Local ID</em>. In other
words, the local ID serves as index into the container which describes what
data stream we want to read from.</p>
-
<h3 id="discovering-the-container-locations">Discovering the Container Locations</h3>
-
<p>How does SCM know where the containers are located ? This is very similar to
what HDFS does; the data nodes regularly send container reports like block
reports. Container reports are far more concise than block reports. For
example, an Ozone deployment with a 196 TB data node will have around 40
thousand containers. Compare that with HDFS block count of million and half
blocks that get reported. That is a 40x reduction in the block reports.</p>
-
<p>This extra indirection helps tremendously with scaling Ozone. SCM has far
less block data to process and the name node is a different service are
critical to scaling Ozone.</p>
diff --git a/docs/0.5.0-beta/concept/hdds.html b/docs/0.5.0-beta/concept/hdds.html
index 96af049..eed7ec5 100644
--- a/docs/0.5.0-beta/concept/hdds.html
+++ b/docs/0.5.0-beta/concept/hdds.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -233,14 +255,12 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Storage container manager provides multiple critical functions for the Ozone
cluster. SCM acts as the cluster manager, Certificate authority, Block
manager and the Replica manager.</p>
-
<div class="col-sm-6">
<div class="media">
@@ -263,7 +283,6 @@ SCM is in charge of creating an Ozone cluster. When an SCM is booted up via <kbd
-
<div class="col-sm-6">
<div class="media">
@@ -290,7 +309,6 @@ token infrastructure depends on this certificate infrastructure.
-
<div class="col-sm-6">
<div class="media">
@@ -315,7 +333,6 @@ read and write these blocks directly.
-
<div class="col-sm-6">
<div class="media">
diff --git a/docs/0.5.0-beta/concept/index.xml b/docs/0.5.0-beta/concept/index.xml
index bd3dfa9..d583550 100644
--- a/docs/0.5.0-beta/concept/index.xml
+++ b/docs/0.5.0-beta/concept/index.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Concepts on Ozone</title>
@@ -6,11 +6,7 @@
<description>Recent content in Concepts on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
- <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/concept/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate><atom:link href="/concept/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Overview</title>
<link>/concept/overview.html</link>
@@ -48,4 +44,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/concept/overview.html b/docs/0.5.0-beta/concept/overview.html
index 3582ded..812642b 100644
--- a/docs/0.5.0-beta/concept/overview.html
+++ b/docs/0.5.0-beta/concept/overview.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Overview</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,61 +255,45 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone is a redundant, distributed object store optimized for Big data
workloads. The primary design point of ozone is scalability, and it aims to
scale to billions of objects.</p>
-
<p>Ozone separates namespace management and block space management; this helps
ozone to scale much better. The namespace is managed by a daemon called
<a href="../concept/ozonemanager.html">Ozone Manager </a> (OM), and block space is
managed by <a href="../concept/hdds.html">Storage Container Manager</a> (SCM).</p>
-
<p>Ozone consists of volumes, buckets, and keys.
A volume is similar to a home directory in the ozone world.
Only an administrator can create it.</p>
-
<p>Volumes are used to store buckets.
Once a volume is created users can create as many buckets as needed.
Ozone stores data as keys which live inside these buckets.</p>
-
<p>Ozone namespace is composed of many storage volumes.
Storage volumes are also used as the basis for storage accounting.</p>
-
<p>The block diagram shows the core components of Ozone.</p>
-
-<p><img src="ozoneBlockDiagram.png" alt="Architecture diagram" /></p>
-
+<p><img src="ozoneBlockDiagram.png" alt="Architecture diagram"></p>
<p>The Ozone Manager is the name space manager, Storage Container Manager
manages the physical and data layer and Recon is the management interface for
Ozone.</p>
-
<h2 id="different-perspectives">Different Perspectives</h2>
-
-<p><img src="FunctionalOzone.png" alt="FunctionalOzone" /></p>
-
+<p><img src="FunctionalOzone.png" alt="FunctionalOzone"></p>
<p>Any distributed system can be viewed from different perspectives. One way to
look at Ozone is to imagine it as Ozone Manager as a name space service built on
- top of HDDS, a distributed block store.</p>
-
+top of HDDS, a distributed block store.</p>
<p>Another way to visualize Ozone is to look at the functional layers; we have a
- metadata data management layer, composed of Ozone Manager and Storage
- Container Manager.</p>
-
+metadata data management layer, composed of Ozone Manager and Storage
+Container Manager.</p>
<p>We have a data storage layer, which is basically the data nodes and they are
- managed by SCM.</p>
-
+managed by SCM.</p>
<p>The replication layer, provided by Ratis is used to replicate metadata (OM and SCM)
and also used for consistency when data is modified at the
data nodes.</p>
-
<p>We have a management server called Recon, that talks to all other components
of Ozone and provides a unified management API and UX for Ozone.</p>
-
<p>We have a protocol bus that allows Ozone to be extended via other
protocols. We currently only have S3 protocol support built via Protocol bus.
Protocol Bus provides a generic notion that you can implement new file system
- or object store protocols that call into O3 Native protocol.</p>
+or object store protocols that call into O3 Native protocol.</p>
diff --git a/docs/0.5.0-beta/concept/ozonemanager.html b/docs/0.5.0-beta/concept/ozonemanager.html
index 595c1e8..06f2221 100644
--- a/docs/0.5.0-beta/concept/ozonemanager.html
+++ b/docs/0.5.0-beta/concept/ozonemanager.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Ozone Manager</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,65 +255,57 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone Manager (OM) is the namespace manager for Ozone.</p>
-
<p>This means that when you want to write some data, you ask Ozone
Manager for a block and Ozone Manager gives you a block and remembers that
information. When you want to read that file back, you need to find the
address of the block and Ozone Manager returns it you.</p>
-
<p>Ozone Manager also allows users to organize keys under a volume and bucket.
Volumes and buckets are part of the namespace and managed by Ozone Manager.</p>
-
<p>Each ozone volume is the root of an independent namespace under OM.
This is very different from HDFS which provides a single rooted file system.</p>
-
<p>Ozone’s namespace is a collection of volumes or is a forest instead of a
single rooted tree as in HDFS. This property makes it easy to deploy multiple
OMs for scaling.</p>
-
<h2 id="ozone-manager-metadata">Ozone Manager Metadata</h2>
-
<p>OM maintains a list of volumes, buckets, and keys.
For each user, it maintains a list of volumes.
For each volume, the list of buckets and for each bucket the list of keys.</p>
-
<p>Ozone Manager will use Apache Ratis(A Raft protocol implementation) to
replicate Ozone Manager state. This will ensure High Availability for Ozone.</p>
-
<h2 id="ozone-manager-and-storage-container-manager">Ozone Manager and Storage Container Manager</h2>
-
<p>The relationship between Ozone Manager and Storage Container Manager is best
understood if we trace what happens during a key write and key read.</p>
-
<h3 id="key-write">Key Write</h3>
-
<ul>
-<li><p>To write a key to Ozone, a client tells Ozone manager that it would like to
+<li>
+<p>To write a key to Ozone, a client tells Ozone manager that it would like to
write a key into a bucket that lives inside a specific volume. Once Ozone
Manager determines that you are allowed to write a key to the specified bucket,
-OM needs to allocate a block for the client to write data.</p></li>
-
-<li><p>To allocate a block, Ozone Manager sends a request to Storage Container
+OM needs to allocate a block for the client to write data.</p>
+</li>
+<li>
+<p>To allocate a block, Ozone Manager sends a request to Storage Container
Manager (SCM); SCM is the manager of data nodes. SCM picks three data nodes
into which client can write data. SCM allocates the block and returns the
-block ID to Ozone Manager.</p></li>
-
-<li><p>Ozone manager records this block information in its metadata and returns the
+block ID to Ozone Manager.</p>
+</li>
+<li>
+<p>Ozone manager records this block information in its metadata and returns the
block and a block token (a security permission to write data to the block)
-to the client.</p></li>
-
-<li><p>The client uses the block token to prove that it is allowed to write data to
-the block and writes data to the data node.</p></li>
-
-<li><p>Once the write is complete on the data node, the client will update the block
+to the client.</p>
+</li>
+<li>
+<p>The client uses the block token to prove that it is allowed to write data to
+the block and writes data to the data node.</p>
+</li>
+<li>
+<p>Once the write is complete on the data node, the client will update the block
information on
-Ozone manager.</p></li>
+Ozone manager.</p>
+</li>
</ul>
-
<h3 id="key-reads">Key Reads</h3>
-
<ul>
<li>Key reads are simpler, the client requests the block list from the Ozone
Manager</li>
diff --git a/docs/0.5.0-beta/design.html b/docs/0.5.0-beta/design.html
index 4ffb877..c2354a4 100644
--- a/docs/0.5.0-beta/design.html
+++ b/docs/0.5.0-beta/design.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -238,8 +260,7 @@
</h2>
- <p class="card-text">title: Decommissioning in Ozone summary: Formal process to shut down machines in a safe way after the required replications. date: 2019-07-31 jira: HDDS-1881 status: current
-author: Anu Engineer, Marton Elek, Stephen O’Donnell Abstract The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.
+ <p class="card-text">title: Decommissioning in Ozone summary: Formal process to shut down machines in a safe way after the required replications. date: 2019-07-31 jira: HDDS-1881 status: current author: Anu Engineer, Marton Elek, Stephen O’Donnell Abstract The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.
There are two main classes of the decommissioning:</p>
<a href="./design/decommissioning.html"
class=" btn btn-primary btn-lg"></a>
diff --git a/docs/0.5.0-beta/design/decommissioning.html b/docs/0.5.0-beta/design/decommissioning.html
index 58b4e63..35b4310 100644
--- a/docs/0.5.0-beta/design/decommissioning.html
+++ b/docs/0.5.0-beta/design/decommissioning.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,32 +235,25 @@
<div class="col-md-9">
<h1></h1>
-
-
-<hr />
-
-<p>title: Decommissioning in Ozone
+ <hr>
+<h2 id="author-anu-engineer-marton-elek-stephen-odonnell">title: Decommissioning in Ozone
summary: Formal process to shut down machines in a safe way after the required replications.
date: 2019-07-31
jira: HDDS-1881
-status: current</p>
-
-<h2 id="author-anu-engineer-marton-elek-stephen-o-donnell">author: Anu Engineer, Marton Elek, Stephen O’Donnell</h2>
-
+status: current
+author: Anu Engineer, Marton Elek, Stephen O’Donnell</h2>
<h1 id="abstract">Abstract</h1>
-
<p>The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.</p>
-
<p>There are two main classes of the decommissioning:</p>
-
<ul>
-<li><p><strong>Maintenance mode</strong>: where the node is expected to be back after a while. It may not require replication of containers if enough replicas are available from other nodes (as we expect to have the current replicas after the restart.)</p></li>
-
-<li><p><strong>Decommissioning</strong>: where the node won’t be started again. All the data should be replicated according to the current replication rules.</p></li>
+<li>
+<p><strong>Maintenance mode</strong>: where the node is expected to be back after a while. It may not require replication of containers if enough replicas are available from other nodes (as we expect to have the current replicas after the restart.)</p>
+</li>
+<li>
+<p><strong>Decommissioning</strong>: where the node won’t be started again. All the data should be replicated according to the current replication rules.</p>
+</li>
</ul>
-
<p>Goals:</p>
-
<ul>
<li>Decommissioning can be canceled any time</li>
<li>The progress of the decommissioning should be trackable</li>
@@ -248,92 +263,61 @@ status: current</p>
<li>The administrator should be notified when a node can be turned off.</li>
<li>The maintenance mode can be time constrained: if the node marked for maintenance for 1 week and the node is not up after one week, the containers should be considered as lost (DEAD node) and should be replicated.</li>
</ul>
-
<h1 id="introduction">Introduction</h1>
-
<p>Ozone is a highly available file system that relies on commodity hardware. In other words, Ozone is designed to handle failures of these nodes all the time.</p>
-
<p>The Storage Container Manager(SCM) is designed to monitor the node health and replicate blocks and containers as needed.</p>
-
<p>At times, Operators of the cluster can help the SCM by giving it hints. When removing a datanode, the operator can provide a hint. That is, a planned failure of the node is coming up, and SCM can make sure it reaches a safe state to handle this planned failure.</p>
-
<p>Some times, this failure is transient; that is, the operator is taking down this node temporarily. In that case, we can live with lower replica counts by being optimistic.</p>
-
<p>Both of these operations, <strong>Maintenance</strong>, and <strong>Decommissioning</strong> are similar from the Replication point of view. In both cases, and the user instructs us on how to handle an upcoming failure.</p>
-
<p>Today, SCM (<em>Replication Manager</em> component inside SCM) understands only one form of failure handling. This paper extends Replica Manager failure modes to allow users to request which failure handling model to be adopted(Optimistic or Pessimistic).</p>
-
<p>Based on physical realities, there are two responses to any perceived failure, to heal the system by taking corrective actions or ignore the failure since the actions in the future will heal the system automatically.</p>
-
<h2 id="user-experiences-decommissioning-vs-maintenance-mode">User Experiences (Decommissioning vs Maintenance mode)</h2>
-
<p>From the user’s point of view, there are two kinds of planned failures that the user would like to communicate to Ozone.</p>
-
<p>The first kind is when a ‘real’ failure is going to happen in the future. This ‘real’ failure is the act of decommissioning. We denote this as “decommission” throughout this paper. The response that the user wants is SCM/Ozone to make replicas to deal with the planned failure.</p>
-
-<p>The second kind is when the failure is ‘transient.’ The user knows that this failure is temporary and cluster in most cases can safely ignore this issue. However, if the transient failures are going to cause a failure of availability; then the user would like the Ozone to take appropriate actions to address it. An example of this case, is if the user put 3 data nodes into maintenance mode and switched them off.</p>
-
+<p>The second kind is when the failure is ’transient.’ The user knows that this failure is temporary and cluster in most cases can safely ignore this issue. However, if the transient failures are going to cause a failure of availability; then the user would like the Ozone to take appropriate actions to address it. An example of this case, is if the user put 3 data nodes into maintenance mode and switched them off.</p>
<p>The transient failure can violate the availability guarantees of Ozone; Since the user is telling us not to take corrective actions. Many times, the user does not understand the impact on availability while asking Ozone to ignore the failure.</p>
-
<p>So this paper proposes the following definitions for Decommission and Maintenance of data nodes.</p>
-
<p><strong>Decommission</strong> <em>of a data node is deemed to be complete when SCM/Ozone completes the replica of all containers on decommissioned data node to other data nodes.That is, the expected count matches the healthy count of containers in the cluster</em>.</p>
-
<p><strong>Maintenance mode</strong> of a data node is complete if Ozone can guarantee at <em>least one copy of every container</em> is available in other healthy data nodes.</p>
-
<h2 id="examples">Examples</h2>
-
<p>Here are some illustrative examples:</p>
-
<ol>
-<li><p>Let us say we have a container, which has only one copy and resides on Machine A. If the user wants to put machine A into maintenance mode; Ozone will make a replica before entering the maintenance mode.</p></li>
-
-<li><p>Suppose a container has two copies, and the user wants to put Machine A to maintenance mode. In this case; the Ozone understands that availability of the container is not affected and hence can decide to forgo replication.</p></li>
-
-<li><p>Suppose a container has two copies, and the user wants to put Machine A into maintenance mode. However, the user wants to put the machine into maintenance mode for one month. As the period of maintenance mode increases, the probability of data loss increases; hence, Ozone might choose to make a replica of the container even if we are entering maintenance mode.</p></li>
-
-<li><p>The semantics of decommissioning means that as long as we can find copies of containers in other machines, we can technically get away with calling decommission complete. Hence this clarification node; in the ordinary course of action; each decommission will create a replication flow for each container we have; however, it is possible to complete a decommission of a data node, even if we get a failure of the data node being decommissioned. As long as we can find the other datanod [...]
-
-<li><p>Let us say we have a copy of a container replica on Machine A, B, and C. It is possible to decommission all three machines at the same time, as decommissioning is just a status indicator of the data node and until we finish the decommissioning process.</p></li>
+<li>
+<p>Let us say we have a container, which has only one copy and resides on Machine A. If the user wants to put machine A into maintenance mode; Ozone will make a replica before entering the maintenance mode.</p>
+</li>
+<li>
+<p>Suppose a container has two copies, and the user wants to put Machine A to maintenance mode. In this case; the Ozone understands that availability of the container is not affected and hence can decide to forgo replication.</p>
+</li>
+<li>
+<p>Suppose a container has two copies, and the user wants to put Machine A into maintenance mode. However, the user wants to put the machine into maintenance mode for one month. As the period of maintenance mode increases, the probability of data loss increases; hence, Ozone might choose to make a replica of the container even if we are entering maintenance mode.</p>
+</li>
+<li>
+<p>The semantics of decommissioning means that as long as we can find copies of containers in other machines, we can technically get away with calling decommission complete. Hence this clarification node; in the ordinary course of action; each decommission will create a replication flow for each container we have; however, it is possible to complete a decommission of a data node, even if we get a failure of the data node being decommissioned. As long as we can find the other datanodes t [...]
+</li>
+<li>
+<p>Let us say we have a copy of a container replica on Machine A, B, and C. It is possible to decommission all three machines at the same time, as decommissioning is just a status indicator of the data node and until we finish the decommissioning process.</p>
+</li>
</ol>
-
<p>The user-visible features for both of these are very similar:</p>
-
<p>Both Decommission and Maintenance mode can be canceled any time before the operation is marked as completed by SCM.</p>
-
<p>Decommissioned nodes, if and when added back, shall be treated as new data nodes; if they have blocks or containers on them, they can be used to reconstruct data.</p>
-
<h2 id="maintenance-mode-in-hdfs">Maintenance mode in HDFS</h2>
-
<p>HDFS supports decommissioning and maintenance mode similar to Ozone. This is a quick description of the HDFS approach.</p>
-
<p>The usage of HDFS maintenance mode:</p>
-
<ul>
<li>First, you set a minimum replica count on the cluster, which can be zero, but defaults to 1.</li>
<li>Then you can set a number of nodes into maintenance, with an expiry time or have them remain in maintenance forever, until they are manually removed. Nodes are put into maintenance in much the same way as nodes are decommissioned.</li>
-<li>When a set of nodes go into maintenance, all blocks hosted on them are scanned and if the node going into maintenance would cause the number of replicas to fall below the minimum replica count, the relevant nodes go into a decommissioning like state while new replicas are made for the blocks.
-
-<ul>
+<li>When a set of nodes go into maintenance, all blocks hosted on them are scanned and if the node going into maintenance would cause the number of replicas to fall below the minimum replica count, the relevant nodes go into a decommissioning like state while new replicas are made for the blocks.</li>
<li>Once the node goes into maintenance, it can be stopped etc and HDFS will not be concerned about the under-replicated state of the blocks.</li>
<li>When the expiry time passes, the node is put back to normal state (if it is online and heartbeating) or marked as dead, at which time new replicas will start to be made.</li>
-</ul></li>
</ul>
-
<p>This is very similar to decommissioning, and the code to track maintenance mode and ensure the blocks are replicated etc, is effectively the same code as with decommissioning. The one area that differs is probably in the replication monitor as it must understand that the node is expected to be offline.</p>
-
<p>The ideal way to use maintenance mode, is when you know there are a set of nodes you can stop without having to do any replications. In HDFS, the rack awareness states that all blocks should be on two racks, so that means a rack can be put into maintenance safely.</p>
-
<p>There is another feature in HDFS called “upgrade Domain” which allows each datanode to be assigned a group. By default there should be at least 3 groups (domains) and then each of the 3 replicas will be stored on different group, allowing one full group to be put into maintenance at once. That is not yet supported in CDH, but is something we are targeting for CDPD I believe.</p>
-
<p>One other difference with maintenance mode and decommissioning, is that you must have some sort of monitor thread checking for when maintenance is scheduled to end. HDFS solves this by having a class called the DatanodeAdminManager, and it tracks all nodes transitioning state, the under-replicated block count on them etc.</p>
-
<h1 id="implementation">Implementation</h1>
-
<h2 id="datanode-state-machine">Datanode state machine</h2>
-
<p><code>NodeStateManager</code> maintains the state of the connected datanodes. The possible states:</p>
-
<table>
<thead>
<tr>
@@ -341,85 +325,72 @@ status: current</p>
<th>description</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>HEALTHY</td>
<td>The node is up and running.</td>
</tr>
-
<tr>
<td>STALE</td>
<td>Some heartbeats were missing for an already missing nodes.</td>
</tr>
-
<tr>
<td>DEAD</td>
<td>The stale node has not been recovered.</td>
</tr>
-
<tr>
<td>ENTERING_MAINTENANCE</td>
<td>The in-progress state, scheduling is disabled but the node can’t not been turned off due to in-progress replication.</td>
</tr>
-
<tr>
<td>IN_MAINTENANCE</td>
<td>Node can be turned off but we expecteed to get it back and have all the replicas.</td>
</tr>
-
<tr>
<td>DECOMMISSIONING</td>
<td>The in-progress state, scheduling is disabled, all the containers should be replicated to other nodes.</td>
</tr>
-
<tr>
<td>DECOMMISSIONED</td>
<td>The node can be turned off, all the containers are replicated to other machine</td>
</tr>
</tbody>
</table>
-
<h2 id="high-level-algorithm">High level algorithm</h2>
-
<p>The Algorithm is pretty simple from the Decommission or Maintenance point of view;</p>
-
<ol>
-<li><p>Mark a data node as DECOMMISSIONING or ENTERING_MAINTENANCE. This implies that node is NOT healthy anymore; we assume the use of a single flag and law of excluded middle.</p></li>
-
-<li><p>Pipelines should be shut down and wait for confirmation that all pipelines are shutdown. So no new I/O or container creation can happen on a Datanode that is part of decomm/maint.</p></li>
-
-<li><p>Once the Node has been marked as DECOMMISSIONING or ENTERING_MAINTENANCE; the Node will generate a list of containers that need replication. This list is generated by the Replica Count decisions for each container; the Replica Count will be computed by Replica Manager;</p></li>
-
-<li><p>Replica Manager will check the stop condition for each node. The following should be true for all the containers to go from DECOMMISSIONING to DECOMMISSIONED or from ENTERING_MAINTENANCE to IN_MAINTENANCE.</p>
-
+<li>
+<p>Mark a data node as DECOMMISSIONING or ENTERING_MAINTENANCE. This implies that node is NOT healthy anymore; we assume the use of a single flag and law of excluded middle.</p>
+</li>
+<li>
+<p>Pipelines should be shut down and wait for confirmation that all pipelines are shutdown. So no new I/O or container creation can happen on a Datanode that is part of decomm/maint.</p>
+</li>
+<li>
+<p>Once the Node has been marked as DECOMMISSIONING or ENTERING_MAINTENANCE; the Node will generate a list of containers that need replication. This list is generated by the Replica Count decisions for each container; the Replica Count will be computed by Replica Manager;</p>
+</li>
+<li>
+<p>Replica Manager will check the stop condition for each node. The following should be true for all the containers to go from DECOMMISSIONING to DECOMMISSIONED or from ENTERING_MAINTENANCE to IN_MAINTENANCE.</p>
+</li>
+</ol>
<ul>
<li>Container is closed.</li>
<li>We have at least one HEALTHY copy at all times.</li>
<li>For entering DECOMMISSIONED mode <code>maintenance + healthy</code> must equal to <code>expectedeCount</code></li>
-</ul></li>
-
-<li><p>We will update the node state to DECOMMISSIONED or IN_MAINTENANCE reached state.</p></li>
+</ul>
+<ol start="5">
+<li>We will update the node state to DECOMMISSIONED or IN_MAINTENANCE reached state.</li>
</ol>
-
<p><em>Replica count</em> is a calculated number which represents the number of <em>missing</em> replicas. The number can be negative in case of an over-replicated container.</p>
-
-<h2 id="calculation-of-the-replica-count-required-replicas">Calculation of the <em>Replica count</em> (required replicas)</h2>
-
-<h3 id="counters-variables">Counters / Variables</h3>
-
+<h2 id="calculation-of-the-_replica-count_-required-replicas">Calculation of the <em>Replica count</em> (required replicas)</h2>
+<h3 id="counters--variables">Counters / Variables</h3>
<p>We have 7 different datanode state, but some of them can be combined. At high level we can group the existing state to three groups:</p>
-
<ul>
<li>healthy state (when the container is available)</li>
<li>maintenance (including IN_MAINTENANCE and ENTERING_MAINTENANCE)</li>
<li>all the others.</li>
</ul>
-
<p>To calculate the required steps (required replication + stop condition) we need counter about the first two categories.</p>
-
<p>Each counters should be calculated per container bases.</p>
-
<table>
<thead>
<tr>
@@ -427,127 +398,92 @@ status: current</p>
<th>Variable (# of containers)</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>HEALTHY</td>
<td><code>healthy</code></td>
</tr>
-
<tr>
<td>STALE + DEAD + DECOMMISSIONED</td>
<td></td>
</tr>
-
<tr>
<td>DECOMMISSIONING</td>
<td></td>
</tr>
-
<tr>
<td>ENTERING_MAINTENANCE + IN_MAINTENANCE</td>
<td><code>maintenance</code></td>
</tr>
</tbody>
</table>
-
<h3 id="the-current-replication-model">The current replication model</h3>
-
<p>The current replication model in SCM/Ozone is very simplistic. We compute the replication count or the number of replications that we need to do as:</p>
-
-<pre><code>Replica count = expectedCount - currentCount
-</code></pre>
-
-<p>In case the <em>Replica count</em> is positive, it means that we need to make more replicas. If the number is negative, it means that we are over replicated and we need to remove some replicas of this container. If the Replica count for a container is zero; it means that we have the expected number of containers in the cluster.</p>
-
+<pre tabindex="0"><code>Replica count = expectedCount - currentCount
+</code></pre><p>In case the <em>Replica count</em> is positive, it means that we need to make more replicas. If the number is negative, it means that we are over replicated and we need to remove some replicas of this container. If the Replica count for a container is zero; it means that we have the expected number of containers in the cluster.</p>
<p>To support idempontent placement strategies we should substract the in-fligt replications from the result: If there are one in-flight replication process and two replicas we won’t start a new replication command unless the original command is timed out. The timeout is configured with <code>hdds.scm.replication.event.timeout</code> and the default value is 10 minutes.</p>
-
<p>More preciously the current algorithm is the following:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">replicaCount <span style="color:#f92672">=</span> expectedCount <span style="color:#f92672">-</span> healthy;
-
-<span style="color:#66d9ef">if</span> (replicaCount <span style="color:#f92672">-</span> inflight <span style="color:#a6e22e">copies</span> <span style="color:#f92672">+</span> inflight <span style="color:#a6e22e">deletes</span>) <span style="color:#f92672">></span> 0 {
- <span style="color:#75715e">// container is over replicated
-</span><span style="color:#75715e"></span>}.<span style="color:#a6e22e">else</span> <span style="color:#66d9ef">if</span> (replicaCount <span style="color:#f92672">-</span> inflight <span style="color:#a6e22e">copies</span> <span style="color:#f92672">+</span> inflight <span style="color:#a6e22e">deletes</span>) <span style="color:#f92672"><</span>0 {
- <span style="color:#75715e">// handle under replicated containers
-</span><span style="color:#75715e"></span>}</code></pre></div>
-<p>The handling of inflight copies and deletes are independent from the decommissioning problem therefore here we focus only on the core mode:</p>
-
-<pre><code>replicaCount = expectedCount - healthy;
-</code></pre>
-
-<h3 id="the-proposed-solution">The proposed solution</h3>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>replicaCount <span style="color:#f92672">=</span> expectedCount <span style="color:#f92672">-</span> healthy<span style="color:#f92672">;</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>replicaCount <span style="color:#f92672">-</span> inflight copies <span style="color:#f92672">+</span> inflight deletes<span style="color:#f92672">)</span> <span style="color:#f92672">></span> <span style="color:#ae81ff">0</span> <span style="color:#f92672">{</span>
+</span></span><span style="display:flex;"><span> <span style="color:#75715e">// container is over replicated
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#f92672">}.</span><span style="color:#a6e22e">else</span> <span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>replicaCount <span style="color:#f92672">-</span> inflight copies <span style="color:#f92672">+</span> inflight deletes<span style="color:#f92672">)</span> <span style="color:#f92672"><</span><span style="color:#ae81ff">0</span> <span style="color: [...]
+</span></span><span style="display:flex;"><span> <span style="color:#75715e">// handle under replicated containers
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#f92672">}</span>
+</span></span></code></pre></div><p>The handling of inflight copies and deletes are independent from the decommissioning problem therefore here we focus only on the core mode:</p>
+<pre tabindex="0"><code>replicaCount = expectedCount - healthy;
+</code></pre><h3 id="the-proposed-solution">The proposed solution</h3>
<p>To support the notion that a user can provide hints to the replication model, we propose to add two variables to the current model.</p>
-
<p>In the new model, we propose to break the <code>currentCount</code> into the two separate groups. That is <em>Healthy nodes</em> and <em>Maintenance nodes</em>. The new model replaces the currentCount with these two separate counts. The following function captures the code that drives the logic of computing Replica counts in the new model. The later section discusses the input and output of this model very extensively.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">/**
-</span><span style="color:#75715e"> * Calculate the number of the missing replicas.
-</span><span style="color:#75715e"> *
-</span><span style="color:#75715e"> * @return the number of the missing replicas.
-</span><span style="color:#75715e"> If it's less than zero, the container is over replicated.
-</span><span style="color:#75715e"> */</span>
-<span style="color:#66d9ef">int</span> <span style="color:#a6e22e">getReplicationCount</span>(<span style="color:#66d9ef">int</span> <span style="color:#a6e22e">expectedCount</span>, <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">healthy</span>, <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">maintenance</span>) {
-
- <span style="color:#75715e">//for over replication, count only with the healthy replicas
-</span><span style="color:#75715e"></span> <span style="color:#66d9ef">if</span> (expectedCount <span style="color:#f92672"><=</span> healthy) {
- <span style="color:#66d9ef">return</span> expectedCount <span style="color:#f92672">-</span> healthy;
- }
-
- replicaCount <span style="color:#f92672">=</span> expectedCount <span style="color:#f92672">-</span> (healthy <span style="color:#f92672">+</span> maintenance);
-
- <span style="color:#75715e">//at least one HEALTHY replicas should be guaranteed!
-</span><span style="color:#75715e"></span> <span style="color:#66d9ef">if</span> (replicaCount <span style="color:#f92672">==</span> 0 <span style="color:#f92672">&&</span> healthy <span style="color:#f92672"><</span> 1) {
- replicaCount <span style="color:#f92672">++</span>;
- }
-
- <span style="color:#75715e">//over replication is already handled. Always return with
-</span><span style="color:#75715e"></span> <span style="color:#75715e">// positive as over replication is already handled
-</span><span style="color:#75715e"></span> <span style="color:#66d9ef">return</span> Math.<span style="color:#a6e22e">max</span>(0, replicaCount);
-}</code></pre></div>
-<p>To understand the reasing behind the two special <code>if</code> condition check the examples below.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">/**
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"> * Calculate the number of the missing replicas.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"> *
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"> * @return the number of the missing replicas.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"> If it's less than zero, the container is over replicated.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"> */</span>
+</span></span><span style="display:flex;"><span><span style="color:#66d9ef">int</span> <span style="color:#a6e22e">getReplicationCount</span><span style="color:#f92672">(</span><span style="color:#66d9ef">int</span> expectedCount<span style="color:#f92672">,</span> <span style="color:#66d9ef">int</span> healthy<span style="color:#f92672">,</span> <span style="color:#66d9ef">int</span> maintenance<span style="color:#f92672">)</span> <span style="color:#f92672">{</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span> <span style="color:#75715e">//for over replication, count only with the healthy replicas
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span> <span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>expectedCount <span style="color:#f92672"><=</span> healthy<span style="color:#f92672">)</span> <span style="color:#f92672">{</span>
+</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">return</span> expectedCount <span style="color:#f92672">-</span> healthy<span style="color:#f92672">;</span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span> replicaCount <span style="color:#f92672">=</span> expectedCount <span style="color:#f92672">-</span> <span style="color:#f92672">(</span>healthy <span style="color:#f92672">+</span> maintenance<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span> <span style="color:#75715e">//at least one HEALTHY replicas should be guaranteed!
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span> <span style="color:#66d9ef">if</span> <span style="color:#f92672">(</span>replicaCount <span style="color:#f92672">==</span> <span style="color:#ae81ff">0</span> <span style="color:#f92672">&&</span> healthy <span style="color:#f92672"><</span> <span style="color:#ae81ff">1</span><span style="color:#f92672">)</span> <span style="color:#f92672">{</span>
+</span></span><span style="display:flex;"><span> replicaCount <span style="color:#f92672">++;</span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672">}</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span> <span style="color:#75715e">//over replication is already handled. Always return with
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span> <span style="color:#75715e">// positive as over replication is already handled
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span> <span style="color:#66d9ef">return</span> Math<span style="color:#f92672">.</span><span style="color:#a6e22e">max</span><span style="color:#f92672">(</span><span style="color:#ae81ff">0</span><span style="color:#f92672">,</span> replicaCount<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">}</span>
+</span></span></code></pre></div><p>To understand the reasing behind the two special <code>if</code> condition check the examples below.</p>
<p>We also need to specify two end condition when the DECOMMISSIONING node can be moved to the DECOMMISSIONED state or the ENTERING_MAINTENANCE mode can be moved to the IN_MAINTENANCE state.</p>
-
<p>The following conditions should be true for all the containers and all the containers on the specific node should be closed.</p>
-
<p><strong>From DECOMMISSIONING to DECOMMISSIONED</strong>:</p>
-
<ul>
<li>There are at least one healthy replica</li>
<li>We have three replicas (both helthy and maintenance)</li>
</ul>
-
<p>Which means that our stop condition can be formalized as:</p>
-
-<pre><code>(healthy >= 1) && (healthy + maintenance >= 3)
-</code></pre>
-
-<p>Both the numbers can be configurable:</p>
-
+<pre tabindex="0"><code>(healthy >= 1) && (healthy + maintenance >= 3)
+</code></pre><p>Both the numbers can be configurable:</p>
<ul>
<li>1 is the minimum number of healthy replicas (<code>decommissioning.minimum.healthy-replicas</code>)</li>
<li>3 is the minimum number of existing replicas (<code>decommissioning.minimum.replicas</code>)</li>
</ul>
-
<p>For example <code>decommissioning.minimum.healthy-replicas</code> can be set to two if administrator would like to survive an additional node failure during the maintenance period.</p>
-
<p><strong>From ENTERING_MAINTENANCE to IN_MAINTENANCE:</strong></p>
-
<ul>
<li>There are at least one healthy replicas</li>
</ul>
-
<p>This is the weaker version of the previous condition:</p>
-
-<pre><code>(healthy >= 1)
-</code></pre>
-
-<h3 id="examples-normal-cases">Examples (normal cases)</h3>
-
+<pre tabindex="0"><code>(healthy >= 1)
+</code></pre><h3 id="examples-normal-cases">Examples (normal cases)</h3>
<p>In this section we show example use cases together with the output of the proposed algorithm</p>
-
<h4 id="all-healthy">All healthy</h4>
-
<table>
<thead>
<tr>
@@ -560,7 +496,6 @@ status: current</p>
<th>repCount</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>HEALTHY</td>
@@ -573,15 +508,10 @@ status: current</p>
</tr>
</tbody>
</table>
-
<p>The container C1 exists on machines A, B , and C. All the container reports tell us that the container is healthy. Running the above algorithm, we get:</p>
-
<p><code>expected - healthy + maint. = 3 - (3 + 0) = 0</code></p>
-
<p>It means, <em>“we don’t need no replication”.</em></p>
-
<h4 id="one-failure">One failure</h4>
-
<table>
<thead>
<tr>
@@ -594,7 +524,6 @@ status: current</p>
<th>repCount</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>HEALTHY</td>
@@ -607,15 +536,10 @@ status: current</p>
</tr>
</tbody>
</table>
-
<p>The machine C has failed, and as a result, the healthy count has gone down from <code>3</code> to <code>2</code>. This means that we need to start one replication flow.</p>
-
<p><code>ReplicaCount = expected - healthy + maint. = 3 - (2 + 0) = 1.</code></p>
-
<p>This means that the new model will handle failure cases just like the current model.</p>
-
<h4 id="one-decommissioning">One decommissioning</h4>
-
<table>
<thead>
<tr>
@@ -628,7 +552,6 @@ status: current</p>
<th>repCount</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>HEALTHY</td>
@@ -641,353 +564,522 @@ status: current</p>
</tr>
</tbody>
</table>
-
<p>In this case, machine C is being decommissioned. Therefore the healthy count has gone down to <code>2</code> , and decommission count is <code>1</code>. Since the</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ReplicaCount = expected - healthy + maint`. we have `1 = 3 - (2 + 0)```," data-lang="ReplicaCount = expected - healthy + maint`. we have `1 = 3 - (2 + 0)```,">this gives us the decommission count implicitly. The trick here is to realize that incrementing decommission automatically causes a decrement in the healthy count, which allows us not to have _de [...]
-
-**Stop condition**: Not that if this containers is the only one on node C, node C can be moved to the DECOMMISSIONED state.
-
-#### Failure + decommissioning
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- HEALTHY | DEAD | DECOMM | 3 | 1 | 0 | 2
-
-
-Here is a case where we have a failure of a data node and a decommission of another data node. In this case, the container C1 needs two replica flows to heal itself. The equation is the same and we get
-
-`ReplicaCount(2) = ExpectecCount(3) - healthy(1)`
-
-The maintenance is still zero so ignored in this equation.
-
-#### 1 failure + 2 decommissioning
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- HEALTHY | DECOMM | DECOMM | 3 | 0 | 0 | 3
-
-In this case, we have one failed data node and two data nodes being decommissioned. We need to get three replica flows in the system. This is achieved by:</code></pre></div>
-<p>ReplicaCount(3) = ExpectedCount(3) - (healthy(0) + maintenance(0))</p>
-
-<pre><code>
-#### Maintenance mode
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- HEALTHY | DEAD | MAINT | 3 | 2 | 1 | 0
-
-This represents the normal maintenance mode, where a single machine is marked as in maintenance mode. This means the following:
-
-</code></pre>
-
-<p>ReplicaCount(0) = ExpectedCount(3) - (healthy(2) + maintenance(1)</p>
-
-<pre><code>
-There are no replica flows since the user has asked us to move a single node into maintenance mode, and asked us explicitly not to worry about the single missing node.
-
-**Stop condition**: Not that if this containers is the only one on node C, node C can be moved to the IN_MAINTENANCE state.
-
-#### Maintenance + decommissioning
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- HEALTHY | DECOMM | MAINT | 3 | 1 | 1 | 1
-
-*This is a fascinating case*; We have one good node; one decommissioned node and one node in maintenance mode. The expected result is that the replica manager will launch one replication flow to compensate for the node that is being decommissioned, and we also expect that there will be no replication for the node in maintenance mode.
-
-</code></pre>
-
-<p>Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1))</p>
-
-<pre><code>So as expected we have one replication flow in the system.
-
-**Stop condition**: Not that if this containers is the only one in the system:
-
- * node C can be moved to the IN_MAINTENANCE state
- * node B can not be decommissioned (we need the three replicas first)
-
-#### Decommissioning all the replicas
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- DECOMM | DECOMM | DECOMM | 3 | 0 | 0 | 3
-
-In this case, we deal with all the data nodes being decommissioned. The number of healthy replicas for this container is 0, and hence:
-
-</code></pre>
-
-<p>replicaCount (3) = expectedCount (3)- (healthy(0) + maintenance(0)).</p>
-
-<pre><code>
-This provides us with all 3 independent replica flows in the system.
-
-#### Decommissioning the one remaining replicas
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- DEAD | DEAD | DECOMM | 3 | 0 | 0 | 3
-
-We have two failed nodes and one node in Decomm. It is the opposite of case Line 5, where we have one failed node and 2 nodes in Decomm. The expected results are the same, we get 3 flows.
-
-#### Total failure
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- DEAD | DEAD | DEAD | 3 | 0 | 0 | 3
-
-This is really an error condition. We have lost all 3 data nodes. The Replica Manager will compute that we need to rebuild 3 replicas, but we might not have a source to rebuild from.
-
-### Last replica is on ENTERING_MAINTENANCE
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- DEAD | MAINT | DEAD | 3 | 0 | 1 | 2
-
-Is also an interesting case; we have lost 2 data nodes; and one node is being marked as Maint. Since we have 2 failed nodes, we need 2 replica flows in the system. However, the maintenance mode cannot be entered, since we will lose lone replica if we do that.
-
-### All maintenance
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|--------|--------|---------|---------|----------
- MAINT | MAINT | MAINT | 3 | 0 | 3 | *1*
-
-This is also a very special case; this is the case where the user is telling us to ignore the peril for all 3 replicas being offline. This means that the system will not be able to get to that container and would lead to potential I/O errors. Ozone will strive to avoid that case; this means that Ozone will hit the “if condition” and discover that we our ReplicCount is 0; since the user asked for it; but we are also going to lose all Replicas. At this point of time, we make a conscious de [...]
-
-**This brings us back to the semantics of Maintenance mode in Ozone**. If going into maintenance mode will not lead to a potential I/O failure, we will enter into the maintenance mode; Otherwise, we will replicate and enter into the maintenance mode after the replication is done. This is just the core replication algorithm, not the complete Decommission or Maintenance mode algorithms, just how the replica manager would behave. Once we define the behavior of Replica Manager, rest of the [...]
-
-Note: this is the case why we need the seconf `if` in the model (numbers in the brackets shows the actual value):
-
-</code></pre>
-
-<p>replicaCount(0) = expectedCount(3) - ( healthy(0) + maintenance(0) );</p>
-
-<p>//at least one HEALTHY replicas should be guaranteed!
- if (replicaCount(0) == 0 && healthy(0) < 1) {
- replicaCount ++;
- }</p>
-
-<pre><code>
-### Over replication
-
-For over-replicated containers Ozone prefers to keep the replicas on the healthy nodes. We delete containers only if we have enough replicas on *healthy* nodes.
-
-</code></pre>
-
-<p>int getReplicationCount(int expectedCount, int healthy, int maintenance) {</p>
-
-<p>//for over replication, count only with the healthy replicas
- if (expectedCount <= healthy) {
- return expectedCount - healthy;
- }</p>
-
-<p>replicaCount = … //calculate missing replicas</p>
-
-<p>//over replication is already handled. Always return with
- // positive as over replication is already handled
- return Math.max(0, replicaCount);
-}</p>
-
-<pre><code>
-Please note that we always assume that the the in-flight deletion are applied and the container is already deleted.
-
-There is a very rare case where the in-flight deletion is timed out (and as a result replication manager would assume the container is not deleted) BUT in the mean-time the container finally deleted. It can be survivied with including the creation timestamp in the ContainerDeleteCommand.
-
-### Over replication examples
-
-#### 4 replicas
-
-
- Node A | Node B | Node C | Node D | expctd | healthy | mainten | repCount
- --------|---------|---------|---------|--------|---------|---------|----------
- HEALTHY | HEALTHY | HEALTHY | HEALTHY | 3 | 4 | 0 | 0
-
-This is an easy case as we have too many replicas we can safely remove on.
-
-</code></pre>
-
-<p>if (expectedCount <= healthy) {
- return expectedCount - healthy
-}</p>
-
-<pre><code>
-#### over replicated with IN_MAINTENANCE
-
-
- Node A | Node B | Node C | Node D | expctd | healthy | mainten | repCount
- --------|---------|---------|---------|--------|---------|---------|----------
- HEALTHY | HEALTHY | HEALTHY | MAINT | 3 | 3 | 1 | 0
-
-
-In this case we will delete the forth replica only after node D is restored and healthy again. (expectedCount is not less than healthy). As the `expectedCount (3) <= healthy (3)` the replicaCount will be calculated as `0`.
-
-#### over replicated with IN_MAINTENANCE
-
-
- Node A | Node B | Node C | Node D | expctd | healthy | mainten | repCount
- --------|---------|---------|---------|--------|---------|---------|----------
- HEALTHY | HEALTHY | MAINT | MAINT | 3 | 2 | 2 | 0
-
-Here we are not over-repliacated as we don't have any healthy nodes. We will calculate the under-replication number as defined in the previous section:
-
-</code></pre>
-
-<p>replicaCount(-1) = expectedCount(3) - ( healthy(2) + maintenance(2) );</p>
-
-<pre><code>
-The main algorithm would return with `replicaCount = -1` but as we return `Math.max(0,replicaCount)` the real response will be 0. Waiting for healthy nodes.
-
-### Handling in-flight replications
-
-In-flight replication requests and deletes are handled by the Replica Manager and the problem is orthogonal to the replication problem, but this section shows that the proposed model is not conflicted with the existing approach.
-
-Let's say we have an under-replicated container and we already selected a new datanode to copy a new replica to that specific node.
-
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount
- --------|---------|---------|--------|---------|---------|----------
- HEALTHY | HEALTHY | (copy) | 3 | 2 | 0 | 1
-
-
-Here the Replication Manager detects that one replica is missing but the real copy shouldn't been requested as it's alread inprogress. ReplicaManager must not select a new datanode based on the ContainerPlacementPolicy implementation as the policy may or may not be idempotent.
-
-For example if the placement policy would select a datanode randomly with each loop we would select a new datanode to replicate to.
-
-To avoid such a situation Replica Manager maintains a list of the in-flight copies (in-memory) on the SCM side. In this list we have all the sent replication requests but they are removed after a given amount of time (10 minutes) by default.
-
-With calculating the in-flight copy as a possible replication the Replication Manger doesn't need to request new replication.
-
-When a datanode is marked to be decommissioned there could be any in-flight replication copy process in that time.
-
- * At datanode we should stop all of the in-flight copy (datanodes should be notified about the DECOMMISSIONING/IN_MAINTENANCE state)
- * We never ask any non-healthy nodes to replicate containers.
- * In SCM, we don't need to do any special action
- * In `ReplicationManager` we already have a map about the inflight replications (`Map<ContainerID, List<InflightAction>>`).
- * During a normal replication the number of in-flight replications are counted as real replication (2 real replicas + 1 inflight replication = replica count 3). During this calculation we need to check the current state of the datanodes and ignore the inflight replication if they are assigned to a node which is in decommissioning state. (Or we should update the inflight map, in case of node state change)
-
-### In-flight examples
-
-#### Maintenance + inflight
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount | copy |
- --------|---------|---------|--------|---------|---------|----------|------|
- HEALTHY | MAINT | copying | 3 | 1 | 1 | 1 | 1 |
-
-Here one have one node ENTERING_MAINTENANCE state, and one replica is missing and already started to be replicated. We don't need to start a new copy and node B can be moved to the IN_MAINTENANCE mode.
-
-</code></pre>
-
-<p>Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1))
-Containers to copy (0) = Replica Count (1) - inflight copies (1)</p>
-
-<pre><code>
-#### Maintenance + inflight
-
- Node A | Node B | Node C | expctd | healthy | mainten | repCount | copy |
- --------|---------|---------|--------|---------|---------|----------|------|
- DECOMM | copying | copying | 3 | 0 | 0 | 3 | 1 |
-
-
-Node A can not be DECOMMISSIONED as we have no HEALTHY replicas at all.
-
-
-## Statefulness
-
-SCM stores all the node state in-memory. After a restart on the SCM side the datanode state can be lost.
-
-**Ozone doesn't guarantee that decommissioning/maintenance mode state survives the SCM restarts!!!**
-
- * If SCM restarts DECOMMISSIONED nodes will not report any more container reports and the nodes won't be registered.
- * ENTERING_MAINTENANCE and DECOMMISSIONING nodes will became HEALTHY again and the decommissioning CLI command should be repeated.
- * IN_MAINTENANCE nodes will become DEAD and all the containers will be replicated.
-
- *Ozone assumes that the maintenance mode is used short-term and SCM is not restarted during this specific period.*
-
-*Reasoning*:
-
-Neither of the node state nor the container state are persisted in SCM side. The decommissioned state can be stored on the SCM side (or on the SCM side and the datanode side) which can provide better user experience (and may be implemented).
-
-But to support maintenance mode after restart all the container information is required to be persisted (which is a too big architectural change).
-
-To make a replication decision replication manager needs the number of healthy replicas (they are reported via heartbeats) AND the number of containers on the node which is in maintenance mode. The later one is not available if the SCM is restarted as the container map exists only in the memory and the node which is turned off can't report any more container reports. Therefore the information about the existing containers on the node which is in the maintenance mode **can't be available'**.
-
-## Throttling
-
-SCM should avoid to request too many replication to live enough network bandwidth for the requests.
-
-Replication Manager can easily throttle the replication requests based on `inflightReplication` map, but this problem is independent from the handling of the decommissioning / maintenance mode because it should be solved for any kind of replication not just for this.
-
-## User interface
-
-The decommissioning and maintenance mode can be administered with a CLI interface.
-
-Required feature:
-
- * Set the state of a datanode (to DECOMMISSIONING or ENTERING_MAINTENANCE)
- * Undo the decommissioning process
- * check the current progress:
- * This can be a table with the nodes, status of the nodes, number of containers, containers under replication and containers which doesn't much the stop condition yet (required replications)
- * All the commands can support topology related filters (eg. display the nodes only for a specific rack or show the status of the nodes of s specific rack)
- * Check current replication information of one specific container (required to debug why the decommissioning is stuck)
-
-## Checks before the decommissioning
-
-Decommissioning is requested via a new RPC call with the help of a new CLI tool. The server should check the current state of the cluster and deny the decommissioning if it's not possible. Possible violations:
-
- * Not enough space to store the new replicas.
- * Not enough node to create all kind of pipelines
-
- In case of any violation, the request will fail, but any of theses rules can be turned off with a next request and the decommissioning can be forced.
-
-## Maintain progress
+<p><code>ReplicaCount = expected - healthy + maint`. we have `1 = 3 - (2 + 0)</code>,</p>
+<p>this gives us the decommission count implicitly. The trick here is to realize that incrementing decommission automatically causes a decrement in the healthy count, which allows us not to have <em>decommission</em> in the equation explicitly.</p>
+<p><strong>Stop condition</strong>: Not that if this containers is the only one on node C, node C can be moved to the DECOMMISSIONED state.</p>
+<h4 id="failure--decommissioning">Failure + decommissioning</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DEAD</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>1</td>
+<td>0</td>
+<td>2</td>
+</tr>
+</tbody>
+</table>
+<p>Here is a case where we have a failure of a data node and a decommission of another data node. In this case, the container C1 needs two replica flows to heal itself. The equation is the same and we get</p>
+<p><code>ReplicaCount(2) = ExpectecCount(3) - healthy(1)</code></p>
+<p>The maintenance is still zero so ignored in this equation.</p>
+<h4 id="1-failure--2-decommissioning">1 failure + 2 decommissioning</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DECOMM</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>In this case, we have one failed data node and two data nodes being decommissioned. We need to get three replica flows in the system. This is achieved by:</p>
+<pre tabindex="0"><code>ReplicaCount(3) = ExpectedCount(3) - (healthy(0) + maintenance(0))
+</code></pre><h4 id="maintenance-mode">Maintenance mode</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DEAD</td>
+<td>MAINT</td>
+<td>3</td>
+<td>2</td>
+<td>1</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>This represents the normal maintenance mode, where a single machine is marked as in maintenance mode. This means the following:</p>
+<pre tabindex="0"><code>ReplicaCount(0) = ExpectedCount(3) - (healthy(2) + maintenance(1)
+</code></pre><p>There are no replica flows since the user has asked us to move a single node into maintenance mode, and asked us explicitly not to worry about the single missing node.</p>
+<p><strong>Stop condition</strong>: Not that if this containers is the only one on node C, node C can be moved to the IN_MAINTENANCE state.</p>
+<h4 id="maintenance--decommissioning">Maintenance + decommissioning</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>DECOMM</td>
+<td>MAINT</td>
+<td>3</td>
+<td>1</td>
+<td>1</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p><em>This is a fascinating case</em>; We have one good node; one decommissioned node and one node in maintenance mode. The expected result is that the replica manager will launch one replication flow to compensate for the node that is being decommissioned, and we also expect that there will be no replication for the node in maintenance mode.</p>
+<pre tabindex="0"><code>Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1))
+</code></pre><p>So as expected we have one replication flow in the system.</p>
+<p><strong>Stop condition</strong>: Not that if this containers is the only one in the system:</p>
+<ul>
+<li>node C can be moved to the IN_MAINTENANCE state</li>
+<li>node B can not be decommissioned (we need the three replicas first)</li>
+</ul>
+<h4 id="decommissioning-all-the-replicas">Decommissioning all the replicas</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DECOMM</td>
+<td>DECOMM</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>In this case, we deal with all the data nodes being decommissioned. The number of healthy replicas for this container is 0, and hence:</p>
+<pre tabindex="0"><code>replicaCount (3) = expectedCount (3)- (healthy(0) + maintenance(0)).
+</code></pre><p>This provides us with all 3 independent replica flows in the system.</p>
+<h4 id="decommissioning-the-one-remaining-replicas">Decommissioning the one remaining replicas</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DEAD</td>
+<td>DEAD</td>
+<td>DECOMM</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>We have two failed nodes and one node in Decomm. It is the opposite of case Line 5, where we have one failed node and 2 nodes in Decomm. The expected results are the same, we get 3 flows.</p>
+<h4 id="total-failure">Total failure</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DEAD</td>
+<td>DEAD</td>
+<td>DEAD</td>
+<td>3</td>
+<td>0</td>
+<td>0</td>
+<td>3</td>
+</tr>
+</tbody>
+</table>
+<p>This is really an error condition. We have lost all 3 data nodes. The Replica Manager will compute that we need to rebuild 3 replicas, but we might not have a source to rebuild from.</p>
+<h3 id="last-replica-is-on-entering_maintenance">Last replica is on ENTERING_MAINTENANCE</h3>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DEAD</td>
+<td>MAINT</td>
+<td>DEAD</td>
+<td>3</td>
+<td>0</td>
+<td>1</td>
+<td>2</td>
+</tr>
+</tbody>
+</table>
+<p>Is also an interesting case; we have lost 2 data nodes; and one node is being marked as Maint. Since we have 2 failed nodes, we need 2 replica flows in the system. However, the maintenance mode cannot be entered, since we will lose lone replica if we do that.</p>
+<h3 id="all-maintenance">All maintenance</h3>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>MAINT</td>
+<td>MAINT</td>
+<td>MAINT</td>
+<td>3</td>
+<td>0</td>
+<td>3</td>
+<td><em>1</em></td>
+</tr>
+</tbody>
+</table>
+<p>This is also a very special case; this is the case where the user is telling us to ignore the peril for all 3 replicas being offline. This means that the system will not be able to get to that container and would lead to potential I/O errors. Ozone will strive to avoid that case; this means that Ozone will hit the “if condition” and discover that we our ReplicCount is 0; since the user asked for it; but we are also going to lose all Replicas. At this point of time, we make a conscious [...]
+<p><strong>This brings us back to the semantics of Maintenance mode in Ozone</strong>. If going into maintenance mode will not lead to a potential I/O failure, we will enter into the maintenance mode; Otherwise, we will replicate and enter into the maintenance mode after the replication is done. This is just the core replication algorithm, not the complete Decommission or Maintenance mode algorithms, just how the replica manager would behave. Once we define the behavior of Replica Manag [...]
+<p>Note: this is the case why we need the seconf <code>if</code> in the model (numbers in the brackets shows the actual value):</p>
+<pre tabindex="0"><code> replicaCount(0) = expectedCount(3) - ( healthy(0) + maintenance(0) );
-We need to show the progress of the decommissioning process per node and cluster-wide. We already have the information about the under replicated containers, but we don't know the numbers of the containers before decommissioning.
-Instead of saving the original number of the required replications before (which is very fragile) we don't provide an absolute progress just the numbers of the remaining replication:
+ //at least one HEALTHY replicas should be guaranteed!
+ if (replicaCount(0) == 0 && healthy(0) < 1) {
+ replicaCount ++;
+ }
+</code></pre><h3 id="over-replication">Over replication</h3>
+<p>For over-replicated containers Ozone prefers to keep the replicas on the healthy nodes. We delete containers only if we have enough replicas on <em>healthy</em> nodes.</p>
+<pre tabindex="0"><code>int getReplicationCount(int expectedCount, int healthy, int maintenance) {
+ //for over replication, count only with the healthy replicas
+ if (expectedCount <= healthy) {
+ return expectedCount - healthy;
+ }
-</code></pre>
+ replicaCount = ... //calculate missing replicas
+ //over replication is already handled. Always return with
+ // positive as over replication is already handled
+ return Math.max(0, replicaCount);
+}
+</code></pre><p>Please note that we always assume that the the in-flight deletion are applied and the container is already deleted.</p>
+<p>There is a very rare case where the in-flight deletion is timed out (and as a result replication manager would assume the container is not deleted) BUT in the mean-time the container finally deleted. It can be survivied with including the creation timestamp in the ContainerDeleteCommand.</p>
+<h3 id="over-replication-examples">Over replication examples</h3>
+<h4 id="4-replicas">4 replicas</h4>
<table>
<thead>
<tr>
-<th>Node</th>
-<th>Status</th>
-<th># containers</th>
-<th>in-progress replications</th>
-<th>required replication</th>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>Node D</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
</tr>
</thead>
-
<tbody>
<tr>
-<td>Node A</td>
-<td>ENTERING_MAINTENANCE</td>
-<td>2837</td>
-<td>12</td>
-<td>402</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>3</td>
+<td>4</td>
+<td>0</td>
+<td>0</td>
</tr>
-
+</tbody>
+</table>
+<p>This is an easy case as we have too many replicas we can safely remove on.</p>
+<pre tabindex="0"><code>if (expectedCount <= healthy) {
+ return expectedCount - healthy
+}
+</code></pre><h4 id="over-replicated-with-in_maintenance">over replicated with IN_MAINTENANCE</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>Node D</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>MAINT</td>
+<td>3</td>
+<td>3</td>
+<td>1</td>
+<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>In this case we will delete the forth replica only after node D is restored and healthy again. (expectedCount is not less than healthy). As the <code>expectedCount (3) <= healthy (3)</code> the replicaCount will be calculated as <code>0</code>.</p>
+<h4 id="over-replicated-with-in_maintenance-1">over replicated with IN_MAINTENANCE</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>Node D</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
<tr>
-<td>Node B</td>
<td>HEALTHY</td>
-<td>1239</td>
+<td>HEALTHY</td>
+<td>MAINT</td>
+<td>MAINT</td>
+<td>3</td>
+<td>2</td>
+<td>2</td>
<td>0</td>
+</tr>
+</tbody>
+</table>
+<p>Here we are not over-repliacated as we don’t have any healthy nodes. We will calculate the under-replication number as defined in the previous section:</p>
+<pre tabindex="0"><code>replicaCount(-1) = expectedCount(3) - ( healthy(2) + maintenance(2) );
+</code></pre><p>The main algorithm would return with <code>replicaCount = -1</code> but as we return <code>Math.max(0,replicaCount)</code> the real response will be 0. Waiting for healthy nodes.</p>
+<h3 id="handling-in-flight-replications">Handling in-flight replications</h3>
+<p>In-flight replication requests and deletes are handled by the Replica Manager and the problem is orthogonal to the replication problem, but this section shows that the proposed model is not conflicted with the existing approach.</p>
+<p>Let’s say we have an under-replicated container and we already selected a new datanode to copy a new replica to that specific node.</p>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>HEALTHY</td>
+<td>(copy)</td>
+<td>3</td>
+<td>2</td>
<td>0</td>
+<td>1</td>
</tr>
-
+</tbody>
+</table>
+<p>Here the Replication Manager detects that one replica is missing but the real copy shouldn’t been requested as it’s alread inprogress. ReplicaManager must not select a new datanode based on the ContainerPlacementPolicy implementation as the policy may or may not be idempotent.</p>
+<p>For example if the placement policy would select a datanode randomly with each loop we would select a new datanode to replicate to.</p>
+<p>To avoid such a situation Replica Manager maintains a list of the in-flight copies (in-memory) on the SCM side. In this list we have all the sent replication requests but they are removed after a given amount of time (10 minutes) by default.</p>
+<p>With calculating the in-flight copy as a possible replication the Replication Manger doesn’t need to request new replication.</p>
+<p>When a datanode is marked to be decommissioned there could be any in-flight replication copy process in that time.</p>
+<ul>
+<li>At datanode we should stop all of the in-flight copy (datanodes should be notified about the DECOMMISSIONING/IN_MAINTENANCE state)</li>
+<li>We never ask any non-healthy nodes to replicate containers.</li>
+<li>In SCM, we don’t need to do any special action
+<ul>
+<li>In <code>ReplicationManager</code> we already have a map about the inflight replications (<code>Map<ContainerID, List<InflightAction>></code>).</li>
+<li>During a normal replication the number of in-flight replications are counted as real replication (2 real replicas + 1 inflight replication = replica count 3). During this calculation we need to check the current state of the datanodes and ignore the inflight replication if they are assigned to a node which is in decommissioning state. (Or we should update the inflight map, in case of node state change)</li>
+</ul>
+</li>
+</ul>
+<h3 id="in-flight-examples">In-flight examples</h3>
+<h4 id="maintenance--inflight">Maintenance + inflight</h4>
+<table>
+<thead>
<tr>
-<td>Node C</td>
-<td>IN_MAINTENANCE</td>
-<td>2348</td>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+<th>copy</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>HEALTHY</td>
+<td>MAINT</td>
+<td>copying</td>
+<td>3</td>
+<td>1</td>
+<td>1</td>
+<td>1</td>
+<td>1</td>
+</tr>
+</tbody>
+</table>
+<p>Here one have one node ENTERING_MAINTENANCE state, and one replica is missing and already started to be replicated. We don’t need to start a new copy and node B can be moved to the IN_MAINTENANCE mode.</p>
+<pre tabindex="0"><code>Replica Count (1) = expectedCount(3) - (healthy(1) + maintenance(1))
+Containers to copy (0) = Replica Count (1) - inflight copies (1)
+</code></pre><h4 id="maintenance--inflight-1">Maintenance + inflight</h4>
+<table>
+<thead>
+<tr>
+<th>Node A</th>
+<th>Node B</th>
+<th>Node C</th>
+<th>expctd</th>
+<th>healthy</th>
+<th>mainten</th>
+<th>repCount</th>
+<th>copy</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>DECOMM</td>
+<td>copying</td>
+<td>copying</td>
+<td>3</td>
<td>0</td>
<td>0</td>
+<td>3</td>
+<td>1</td>
</tr>
</tbody>
</table>
-
-<p>```</p>
-
-<p><code># containers</code> means the total number of the containers on the specific datanodes. To get the original number of the planned copies we can save the original ‘container-to-node’ map in memory and show some progress and provide more information for the users.</p>
+<p>Node A can not be DECOMMISSIONED as we have no HEALTHY replicas at all.</p>
+<h2 id="statefulness">Statefulness</h2>
+<p>SCM stores all the node state in-memory. After a restart on the SCM side the datanode state can be lost.</p>
+<p><strong>Ozone doesn’t guarantee that decommissioning/maintenance mode state survives the SCM restarts!!!</strong></p>
+<ul>
+<li>If SCM restarts DECOMMISSIONED nodes will not report any more container reports and the nodes won’t be registered.</li>
+<li>ENTERING_MAINTENANCE and DECOMMISSIONING nodes will became HEALTHY again and the decommissioning CLI command should be repeated.</li>
+<li>IN_MAINTENANCE nodes will become DEAD and all the containers will be replicated.</li>
+</ul>
+<p><em>Ozone assumes that the maintenance mode is used short-term and SCM is not restarted during this specific period.</em></p>
+<p><em>Reasoning</em>:</p>
+<p>Neither of the node state nor the container state are persisted in SCM side. The decommissioned state can be stored on the SCM side (or on the SCM side and the datanode side) which can provide better user experience (and may be implemented).</p>
+<p>But to support maintenance mode after restart all the container information is required to be persisted (which is a too big architectural change).</p>
+<p>To make a replication decision replication manager needs the number of healthy replicas (they are reported via heartbeats) AND the number of containers on the node which is in maintenance mode. The later one is not available if the SCM is restarted as the container map exists only in the memory and the node which is turned off can’t report any more container reports. Therefore the information about the existing containers on the node which is in the maintenance mode <strong>can& [...]
+<h2 id="throttling">Throttling</h2>
+<p>SCM should avoid to request too many replication to live enough network bandwidth for the requests.</p>
+<p>Replication Manager can easily throttle the replication requests based on <code>inflightReplication</code> map, but this problem is independent from the handling of the decommissioning / maintenance mode because it should be solved for any kind of replication not just for this.</p>
+<h2 id="user-interface">User interface</h2>
+<p>The decommissioning and maintenance mode can be administered with a CLI interface.</p>
+<p>Required feature:</p>
+<ul>
+<li>Set the state of a datanode (to DECOMMISSIONING or ENTERING_MAINTENANCE)</li>
+<li>Undo the decommissioning process</li>
+<li>check the current progress:
+<ul>
+<li>This can be a table with the nodes, status of the nodes, number of containers, containers under replication and containers which doesn’t much the stop condition yet (required replications)</li>
+</ul>
+</li>
+<li>All the commands can support topology related filters (eg. display the nodes only for a specific rack or show the status of the nodes of s specific rack)</li>
+<li>Check current replication information of one specific container (required to debug why the decommissioning is stuck)</li>
+</ul>
+<h2 id="checks-before-the-decommissioning">Checks before the decommissioning</h2>
+<p>Decommissioning is requested via a new RPC call with the help of a new CLI tool. The server should check the current state of the cluster and deny the decommissioning if it’s not possible. Possible violations:</p>
+<ul>
+<li>Not enough space to store the new replicas.</li>
+<li>Not enough node to create all kind of pipelines</li>
+</ul>
+<p>In case of any violation, the request will fail, but any of theses rules can be turned off with a next request and the decommissioning can be forced.</p>
+<h2 id="maintain-progress">Maintain progress</h2>
+<p>We need to show the progress of the decommissioning process per node and cluster-wide. We already have the information about the under replicated containers, but we don’t know the numbers of the containers before decommissioning.</p>
+<p>Instead of saving the original number of the required replications before (which is very fragile) we don’t provide an absolute progress just the numbers of the remaining replication:</p>
+<pre tabindex="0"><code> Node | Status | # containers | in-progress replications| required replication
+ ----------------|------------------------|--------------|--------------------------|------------------------
+ Node A | ENTERING_MAINTENANCE | 2837 | 12 | 402
+ Node B | HEALTHY | 1239 | 0 | 0
+ Node C | IN_MAINTENANCE | 2348 | 0 | 0
+</code></pre><p><code># containers</code> means the total number of the containers on the specific datanodes. To get the original number of the planned copies we can save the original ‘container-to-node’ map in memory and show some progress and provide more information for the users.</p>
diff --git a/docs/0.5.0-beta/design/index.xml b/docs/0.5.0-beta/design/index.xml
index bffbe89..89260b1 100644
--- a/docs/0.5.0-beta/design/index.xml
+++ b/docs/0.5.0-beta/design/index.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Designs on Ozone</title>
@@ -6,11 +6,7 @@
<description>Recent content in Designs on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
- <lastBuildDate>Fri, 07 Jun 2019 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/design/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <lastBuildDate>Fri, 07 Jun 2019 00:00:00 +0000</lastBuildDate><atom:link href="/design/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Ozone Enhancement Proposals</title>
<link>/design/ozone-enhancement-proposals.html</link>
@@ -26,10 +22,9 @@
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/design/decommissioning.html</guid>
- <description>title: Decommissioning in Ozone summary: Formal process to shut down machines in a safe way after the required replications. date: 2019-07-31 jira: HDDS-1881 status: current
-author: Anu Engineer, Marton Elek, Stephen O&rsquo;Donnell Abstract The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.
+ <description>title: Decommissioning in Ozone summary: Formal process to shut down machines in a safe way after the required replications. date: 2019-07-31 jira: HDDS-1881 status: current author: Anu Engineer, Marton Elek, Stephen O&rsquo;Donnell Abstract The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.
There are two main classes of the decommissioning:</description>
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/design/ozone-enhancement-proposals.html b/docs/0.5.0-beta/design/ozone-enhancement-proposals.html
index a8cbba3..f69c966 100644
--- a/docs/0.5.0-beta/design/ozone-enhancement-proposals.html
+++ b/docs/0.5.0-beta/design/ozone-enhancement-proposals.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Ozone Enhancement Proposals</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,220 +251,174 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<h2 id="problem-statement">Problem statement</h2>
-
<p>Some of the biggers features requires well defined plans before the implementation. Until now it was managed by uploading PDF design docs to selected JIRA. There are multiple problems with the current practice.</p>
-
<ol>
<li>There is no easy way to find existing up-to-date and outdated design docs.</li>
<li>Design docs usually have better description of the problem that the user docs</li>
<li>We need better tools to discuss the design docs in the development phase of the doc</li>
</ol>
-
<p>We propose to follow the same process what we have now, but instead of uploading a PDF to the JIRA, create a PR to merge the proposal document to the documentation project.</p>
-
<h2 id="non-goals">Non-goals</h2>
-
<ul>
<li>Modify the existing workflow or approval process</li>
<li>Migrate existing documents</li>
<li>Make it harder to create design docs (it should be easy to support the creation of proposals for any kind of tasks)</li>
<li>Define how the design docs are handled/created <em>before</em> the publication (this proposal is about the publishing process)</li>
</ul>
-
<h2 id="proposed-solution">Proposed solution</h2>
-
<ul>
<li>Open a dedicated Jira (<code>HDDS-*</code> but with specific component)</li>
<li>Use standard name prefix in the jira (easy to filter on the mailing list) `[OEP]</li>
<li>Create a PR to add the design doc to the current documentation
-
<ul>
<li>The content of the design can be added to the documentation (Recommended)</li>
<li>Or can be added as external reference</li>
-</ul></li>
+</ul>
+</li>
<li>The design doc (or the summary with the reference) will be merged to the design doc folder of <code>hadoop-hdds/docs/content/design</code> (will be part of the docs)</li>
<li>Discuss it as before (lazy consesus, except if somebody calls for a real vote)</li>
<li>Design docs can be updated according to the changes during the implementation</li>
<li>Only the implemented design docs will be visible as part of the design docs</li>
</ul>
-
<p>As a result all the design docs can be listed under the documentation page.</p>
-
<p>A good design doc has the following properties:</p>
-
<ol>
<li>Publicly available for anybody (Please try to avoid services which are available only with registration, eg: google docs)</li>
<li>Archived for the future (Commit it to the source OR use apache jira or wiki)</li>
<li>Editable later (Best format is markdown, RTF is also good. PDF has a limitation, it’s very hard to reuse the text, or create an updated design doc)</li>
<li>Well structured to make it easy to comment any part of the document (Markdown files which are part of the pull request can be commented in the PR line by line)</li>
</ol>
-
<h3 id="example-1-design-doc-as-a-markdown-file">Example 1: Design doc as a markdown file</h3>
-
<p>The easiest way to create a design doc is to create a new markdown file in a PR and merge it to <code>hadoop-hdds/docs/content/design</code>.</p>
-
<ol>
<li>Publicly available: YES, it can be linked from Apache git or github</li>
<li>Archived: YES, and it’s also versioned. All the change history can be tracked.</li>
<li>Editable later: YES, as it’s just a simple text file</li>
<li>Commentable: YES, comment can be added to each line.</li>
</ol>
-
<h3 id="example-2-design-doc-as-a-pdf">Example 2: Design doc as a PDF</h3>
-
<p>A very common practice of today is to create design doc on google docs and upload it to the JIRA.</p>
-
<ol>
<li>Publicy available: YES, anybody can download it from the Jira.</li>
<li>Archived: YES, it’s available from Apache infra.</li>
<li>Editable: NO, It’s harder to reuse the text to import to the docs or create a new design doc.</li>
<li>Commentable: PARTIAL, Not as easy as a text file or the original google docs, but a good structure with numbered section may help</li>
</ol>
-
<h3 id="the-format">The format</h3>
-
<p>While the first version (markdown files) are the most powerful, the second version (the existing practice) is also acceptable. In this case we propose to create a PR with adding a reference page <em>without</em> the content but including the link.</p>
-
<p>For example:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">---
-title: Ozone Security Design
-summary: A comprehensive description of the security flow between server and client components.
-date: <span style="color:#e6db74">2018-02-22</span>
-jira: HDDS<span style="color:#ae81ff">-4</span>
-status: implemented
-author: Sanjay Radia, Jitendra Pandey, Xiaoyu Yao, Anu Engineer
-
-<span style="color:#75715e">## Summary</span>
-
-Ozone security model is based on Kerberos and similar to the Hadoop security but some of the parts are improved: for example the SCM works as a Certificate Authority and PKI based solutions are wildely used.
-
-<span style="color:#75715e">## Reference</span>
-
-For more details please check the (uploaded design doc)[https://issues.apache.org/jira/secure/attachment/<span style="color:#ae81ff">12911638</span>/HadoopStorageLayerSecurity.pdf].</code></pre></div>
-<p>Obviously with the first approach the design doc itself can be included in this markdown file.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>---
+</span></span><span style="display:flex;"><span><span style="color:#f92672">title</span>: <span style="color:#ae81ff">Ozone Security Design</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">summary</span>: <span style="color:#ae81ff">A comprehensive description of the security flow between server and client components.</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">date</span>: <span style="color:#e6db74">2018-02-22</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">jira</span>: <span style="color:#ae81ff">HDDS-4</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">status</span>: <span style="color:#ae81ff">implemented</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">author</span>: <span style="color:#ae81ff">Sanjay Radia, Jitendra Pandey, Xiaoyu Yao, Anu Engineer</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">## Summary</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">Ozone security model is based on Kerberos and similar to the Hadoop security but some of the parts are improved</span>: <span style="color:#ae81ff">for example the SCM works as a Certificate Authority and PKI based solutions are wildely used.</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">## Reference</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#ae81ff">For more details please check the (uploaded design doc)[https://issues.apache.org/jira/secure/attachment/12911638/HadoopStorageLayerSecurity.pdf].</span>
+</span></span></code></pre></div><p>Obviously with the first approach the design doc itself can be included in this markdown file.</p>
<h2 id="migration">Migration</h2>
-
<p>It’s not a hard requirement to migrate all the design doc. But process is always open:</p>
-
<ol>
<li>To create reference pages for any of the old design docs</li>
<li>To migrate any new design docs to markdown formats (by anybody not just by the author)</li>
<li>To update any of the old design docs based on the current state of the code (We have versioning!)</li>
</ol>
-
<h2 id="document-template">Document template</h2>
-
<p>This the proposed template to document any proposal. It’s recommended but not required the use exactly the some structure. Some proposal may require different structure, but we need the following information.</p>
-
<ol>
<li>Summary</li>
</ol>
-
<blockquote>
<p>Give a one sentence summary, like the jira title. It will be displayed on the documentation page. Should be enough to understand</p>
</blockquote>
-
-<ol>
+<ol start="2">
<li>Status</li>
</ol>
-
<p>Defined in the markdown header. Proposed statuses:</p>
-
<ul>
-<li><p><code>accepted</code>: (Use this as by default. If not accapted, won’t be merged)</p></li>
-
-<li><p><code>implemented</code>: The discussed technical solution is implemented (maybe with some minor implementation difference)</p></li>
-
-<li><p><code>replaced</code>: Replaced by a new design doc</p></li>
-
-<li><p><code>outdated</code>: Code has been changed and design doc doesn’t reflect any more the state of the current code.</p></li>
+<li>
+<p><code>accepted</code>: (Use this as by default. If not accapted, won’t be merged)</p>
+</li>
+<li>
+<p><code>implemented</code>: The discussed technical solution is implemented (maybe with some minor implementation difference)</p>
+</li>
+<li>
+<p><code>replaced</code>: Replaced by a new design doc</p>
+</li>
+<li>
+<p><code>outdated</code>: Code has been changed and design doc doesn’t reflect any more the state of the current code.</p>
+</li>
</ul>
-
<p>Note: the <em>accepted</em> design docs won’t be visible as part of the documentation or only under a dedicated section to clearly comminucate that it’s not ready, yet.</p>
-
-<ol>
+<ol start="3">
<li>Problem statement (Motivation / Abstract)</li>
</ol>
-
<blockquote>
<p>What is the problem and how would you solve it? Think about an abstract of a paper: one paragraph overview. Why will the world better with this change?</p>
</blockquote>
-
-<ol>
+<ol start="4">
<li>Non-goals</li>
</ol>
-
<blockquote>
<p>Very important to define what is outside of the scope of this proposal</p>
</blockquote>
-
-<ol>
+<ol start="5">
<li>Technical Description (Architecture and implementation details)</li>
</ol>
-
<blockquote>
<p>Explain the problem in more details. How can it be reproduced? What is the current solution? What is the limitation of the current solution?</p>
-
+</blockquote>
+<blockquote>
<p>How the new proposed solution would solve the problem? Architectural design.</p>
-
+</blockquote>
+<blockquote>
<p>Implementation details. What should be changed in the code. Is it a huge change? Do we need to change wire protocol? Backward compatibility?</p>
</blockquote>
-
-<ol>
+<ol start="6">
<li>Alternatives</li>
</ol>
-
<blockquote>
<p>What are the other alternatives you considered and why do yoy prefer the proposed solution The goal of this section is to help people understand why this is the best solution now, and also to prevent churn in the future when old alternatives are reconsidered.</p>
</blockquote>
-
-<p>Note: In some cases <sup>4</sup>⁄<sub>5</sub> can be combined. For example if you have multiple proposals, the first version may include multiple solutions. At the end ot the discussion we can move the alternatives to 5. and explain why the community is decided to use the selected option.</p>
-
-<ol>
+<p>Note: In some cases 4/5 can be combined. For example if you have multiple proposals, the first version may include multiple solutions. At the end ot the discussion we can move the alternatives to 5. and explain why the community is decided to use the selected option.</p>
+<ol start="7">
<li>Plan</li>
</ol>
-
<blockquote>
<p>Planning to implement the feature. Estimated size of the work? Do we need feature branch? Any migration plan, dependency? If it’s not a big new feature it can be one sentence or optional.</p>
</blockquote>
-
-<ol>
+<ol start="8">
<li>References</li>
</ol>
-
<h2 id="workflows-form-other-projects">Workflows form other projects</h2>
-
<p>There are similar process in other open source projects. This document and the template is inspired by the following projects:</p>
-
<ul>
<li><a href="https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals">Apache Kafka Improvement Proposals</a></li>
<li><a href="https://spark.apache.org/improvement-proposals.html">Apache Spark Project Improvement Proposals</a></li>
<li><a href="https://github.com/kubernetes/enhancements/tree/master/keps">Kubernetes Enhancement Proposals</a></li>
</ul>
-
<p>Short summary of the processes:</p>
-
<p><strong>Kafka</strong> process:</p>
-
<ul>
<li>Create wiki page</li>
<li>Start discussion on mail thread</li>
<li>Vote on mail thread</li>
</ul>
-
<p><strong>Spark</strong> process:</p>
-
<ul>
<li>Create JIRA (dedicated label)</li>
<li>Discuss on the jira page</li>
<li>Vote on dev list</li>
</ul>
-
<p><em>Kubernetes</em>:</p>
-
<ul>
<li>Deditaced git repository</li>
<li>KEPs are committed to the repo</li>
diff --git a/docs/0.5.0-beta/en/index.html b/docs/0.5.0-beta/en/index.html
index 43042c6..df650c8 100644
--- a/docs/0.5.0-beta/en/index.html
+++ b/docs/0.5.0-beta/en/index.html
@@ -1 +1,10 @@
-<!DOCTYPE html><html><head><title></title><link rel="canonical" href=""/><meta name="robots" content="noindex"><meta charset="utf-8" /><meta http-equiv="refresh" content="0; url=" /></head></html>
\ No newline at end of file
+<!DOCTYPE html>
+<html lang="en-us">
+ <head>
+ <title></title>
+ <link rel="canonical" href="">
+ <meta name="robots" content="noindex">
+ <meta charset="utf-8">
+ <meta http-equiv="refresh" content="0; url=">
+ </head>
+</html>
diff --git a/docs/0.5.0-beta/en/sitemap.xml b/docs/0.5.0-beta/en/sitemap.xml
index 32870bd..fea6260 100644
--- a/docs/0.5.0-beta/en/sitemap.xml
+++ b/docs/0.5.0-beta/en/sitemap.xml
@@ -1,10 +1,9 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
-
<url>
<loc>/</loc>
- <lastmod>2019-06-07T00:00:00+00:00</lastmod>
+ <lastmod>2020-01-07T03:42:43+08:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -15,11 +14,9 @@
hreflang="en"
href="/"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/concept/overview.html</loc>
- <lastmod>2017-10-10T00:00:00+00:00</lastmod>
+ <lastmod>2020-01-06T12:19:26+01:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -30,11 +27,9 @@
hreflang="en"
href="/concept/overview.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/interface/javaapi.html</loc>
- <lastmod>2017-09-14T00:00:00+00:00</lastmod>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -45,10 +40,9 @@
hreflang="en"
href="/interface/javaapi.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/start.html</loc>
+ <lastmod>2019-06-28T18:59:33-04:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -59,18 +53,15 @@
hreflang="en"
href="/start.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/beyond/runningwithhdfs.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-11-07T10:16:50+01:00</lastmod>
+ </url><url>
<loc>/security/secureozone.html</loc>
- </url>
-
- <url>
+ <lastmod>2020-01-06T12:40:46-06:00</lastmod>
+ </url><url>
<loc>/shell/format.html</loc>
+ <lastmod>2020-01-23T14:18:39+08:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -81,11 +72,9 @@
hreflang="en"
href="/shell/format.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/interface/ozonefs.html</loc>
- <lastmod>2017-09-14T00:00:00+00:00</lastmod>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -96,11 +85,9 @@
hreflang="en"
href="/interface/ozonefs.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/concept/ozonemanager.html</loc>
- <lastmod>2017-09-14T00:00:00+00:00</lastmod>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -111,18 +98,15 @@
hreflang="en"
href="/concept/ozonemanager.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/beyond/containers.html</loc>
- </url>
-
- <url>
+ <lastmod>2020-01-06T12:19:26+01:00</lastmod>
+ </url><url>
<loc>/security/securingdatanodes.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
+ </url><url>
<loc>/shell/volumecommands.html</loc>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -133,11 +117,9 @@
hreflang="en"
href="/shell/volumecommands.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/concept/hdds.html</loc>
- <lastmod>2017-09-14T00:00:00+00:00</lastmod>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -148,10 +130,9 @@
hreflang="en"
href="/concept/hdds.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/shell/bucketcommands.html</loc>
+ <lastmod>2020-01-23T16:55:06-08:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -162,10 +143,9 @@
hreflang="en"
href="/shell/bucketcommands.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/interface/s3.html</loc>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -176,15 +156,12 @@
hreflang="en"
href="/interface/s3.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/security/securingtde.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
+ </url><url>
<loc>/concept/datanodes.html</loc>
- <lastmod>2017-09-14T00:00:00+00:00</lastmod>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -195,15 +172,12 @@
hreflang="en"
href="/concept/datanodes.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/beyond/dockercheatsheet.html</loc>
- <lastmod>2017-08-10T00:00:00+00:00</lastmod>
- </url>
-
- <url>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
+ </url><url>
<loc>/shell/keycommands.html</loc>
+ <lastmod>2020-01-25T00:54:46+08:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -214,18 +188,15 @@
hreflang="en"
href="/shell/keycommands.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/security/securings3.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
+ </url><url>
<loc>/security/secuitywithranger.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
+ </url><url>
<loc>/gdpr.html</loc>
+ <lastmod>2019-09-19T10:58:01+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -236,10 +207,9 @@
hreflang="en"
href="/gdpr.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/gdpr/gdpr-in-ozone.html</loc>
+ <lastmod>2019-09-19T10:58:01+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -250,15 +220,12 @@
hreflang="en"
href="/gdpr/gdpr-in-ozone.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/security.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-09-19T16:41:00-07:00</lastmod>
+ </url><url>
<loc>/concept.html</loc>
- <lastmod>2017-10-10T00:00:00+00:00</lastmod>
+ <lastmod>2019-06-28T19:51:30+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -269,24 +236,18 @@
hreflang="en"
href="/concept.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/security/securityacls.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-09-19T16:41:00-07:00</lastmod>
+ </url><url>
<loc>/beyond.html</loc>
- <lastmod>2017-10-10T00:00:00+00:00</lastmod>
- </url>
-
- <url>
+ <lastmod>2019-06-28T19:51:30+02:00</lastmod>
+ </url><url>
<loc>/recipe.html</loc>
- <lastmod>2017-10-10T00:00:00+00:00</lastmod>
- </url>
-
- <url>
+ <lastmod>2019-09-19T14:42:33+02:00</lastmod>
+ </url><url>
<loc>/start/startfromdockerhub.html</loc>
+ <lastmod>2019-09-16T15:41:17+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -297,10 +258,9 @@
hreflang="en"
href="/start/startfromdockerhub.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/start/onprem.html</loc>
+ <lastmod>2019-10-30T09:34:28-07:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -311,10 +271,9 @@
hreflang="en"
href="/start/onprem.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/start/minikube.html</loc>
+ <lastmod>2019-09-16T15:41:17+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -325,10 +284,9 @@
hreflang="en"
href="/start/minikube.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/start/kubernetes.html</loc>
+ <lastmod>2019-09-16T15:41:17+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -339,10 +297,9 @@
hreflang="en"
href="/start/kubernetes.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/start/runningviadocker.html</loc>
+ <lastmod>2019-09-16T15:41:17+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -353,10 +310,9 @@
hreflang="en"
href="/start/runningviadocker.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/start/fromsource.html</loc>
+ <lastmod>2019-10-14T10:01:24+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
@@ -367,82 +323,61 @@
hreflang="en"
href="/start/fromsource.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/design.html</loc>
- <lastmod>2019-06-07T00:00:00+00:00</lastmod>
- </url>
-
- <url>
+ <lastmod>2019-08-28T22:28:07+02:00</lastmod>
+ </url><url>
<loc>/design/ozone-enhancement-proposals.html</loc>
- <lastmod>2019-06-07T00:00:00+00:00</lastmod>
- </url>
-
- <url>
+ <lastmod>2019-08-14T02:10:36+02:00</lastmod>
+ </url><url>
<loc>/tools/genconf.html</loc>
- <lastmod>2018-12-18T00:00:00+00:00</lastmod>
- </url>
-
- <url>
+ <lastmod>2019-09-19T14:42:33+02:00</lastmod>
+ </url><url>
<loc>/tools/auditparser.html</loc>
- <lastmod>2018-12-17T00:00:00+00:00</lastmod>
- </url>
-
- <url>
+ <lastmod>2019-09-19T14:42:33+02:00</lastmod>
+ </url><url>
<loc>/tools.html</loc>
- <lastmod>2017-10-10T00:00:00+00:00</lastmod>
- </url>
-
- <url>
- <loc>/interface.html</loc>
- <lastmod>2017-09-14T00:00:00+00:00</lastmod>
+ <lastmod>2019-09-19T14:42:33+02:00</lastmod>
+ </url><url>
+ <loc>/tools/scmcli.html</loc>
+ <lastmod>2019-09-19T14:42:33+02:00</lastmod>
+ </url><url>
+ <loc>/design/decommissioning.html</loc>
+ <lastmod>2019-08-28T22:28:07+02:00</lastmod>
+ </url><url>
+ <loc>/shell.html</loc>
+ <lastmod>2019-06-28T19:51:30+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
- href="/zh/interface.html"
+ href="/zh/shell.html"
/>
<xhtml:link
rel="alternate"
hreflang="en"
- href="/interface.html"
+ href="/shell.html"
/>
- </url>
-
- <url>
- <loc>/tools/scmcli.html</loc>
- <lastmod>2017-08-10T00:00:00+00:00</lastmod>
- </url>
-
- <url>
- <loc>/design/decommissioning.html</loc>
- <priority>0</priority>
- </url>
-
- <url>
- <loc>/shell.html</loc>
+ </url><url>
+ <loc>/interface.html</loc>
+ <lastmod>2019-06-28T19:51:30+02:00</lastmod>
<xhtml:link
rel="alternate"
hreflang="zh"
- href="/zh/shell.html"
+ href="/zh/interface.html"
/>
<xhtml:link
rel="alternate"
hreflang="en"
- href="/shell.html"
+ href="/interface.html"
/>
- </url>
-
- <url>
+ </url><url>
<loc>/recipe/prometheus.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
+ </url><url>
<loc>/recipe/sparkozonefsk8s.html</loc>
- </url>
-
- <url>
+ <lastmod>2019-08-21T22:47:41+05:30</lastmod>
+ </url><url>
<loc>/tools/testtools.html</loc>
+ <lastmod>2019-09-19T14:42:33+02:00</lastmod>
</url>
-
-</urlset>
\ No newline at end of file
+</urlset>
diff --git a/docs/0.5.0-beta/gdpr.html b/docs/0.5.0-beta/gdpr.html
index d29da26..2184164 100644
--- a/docs/0.5.0-beta/gdpr.html
+++ b/docs/0.5.0-beta/gdpr.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -225,7 +247,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">GDPR compliance in Ozone </h3>
@@ -237,12 +258,10 @@
</p>
</div>
</div>
-
<div class="alert alert-warning" role="alert">
If you would like to understand Ozone's GDPR framework at a greater
depth, please take a look at <a href="https://issues.apache.org/jira/secure/attachment/12978992/Ozone%20GDPR%20Framework.pdf">Ozone GDPR Framework.</a>
</div>
-
<p>Once you create a GDPR compliant bucket, any key created in that bucket will
automatically by GDPR compliant.</p>
diff --git a/docs/0.5.0-beta/gdpr/gdpr-in-ozone.html b/docs/0.5.0-beta/gdpr/gdpr-in-ozone.html
index 02ea541..f6f5296 100644
--- a/docs/0.5.0-beta/gdpr/gdpr-in-ozone.html
+++ b/docs/0.5.0-beta/gdpr/gdpr-in-ozone.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -233,22 +255,16 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Enabling GDPR compliance in Ozone is very straight forward. During bucket
creation, you can specify <code>--enforcegdpr=true</code> or <code>-g=true</code> and this will
ensure the bucket is GDPR compliant. Thus, any key created under this bucket
will automatically be GDPR compliant.</p>
-
<p>GDPR can only be enabled on a new bucket. For existing buckets, you would
have to create a new GDPR compliant bucket and copy data from old bucket into
- new bucket to take advantage of GDPR.</p>
-
+new bucket to take advantage of GDPR.</p>
<p>Example to create a GDPR compliant bucket:</p>
-
<p><code>ozone sh bucket create --enforcegdpr=true /hive/jan</code></p>
-
<p><code>ozone sh bucket create -g=true /hive/jan</code></p>
-
<p>If you want to create an ordinary bucket then you can skip <code>--enforcegdpr</code>
and <code>-g</code> flags.</p>
diff --git a/docs/0.5.0-beta/gdpr/index.xml b/docs/0.5.0-beta/gdpr/index.xml
index 4cede75..e6489d7 100644
--- a/docs/0.5.0-beta/gdpr/index.xml
+++ b/docs/0.5.0-beta/gdpr/index.xml
@@ -1,15 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>GDPR on Ozone</title>
<link>/gdpr.html</link>
<description>Recent content in GDPR on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
-
- <atom:link href="/gdpr/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/gdpr/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>GDPR in Ozone</title>
<link>/gdpr/gdpr-in-ozone.html</link>
@@ -20,4 +16,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/index.html b/docs/0.5.0-beta/index.html
index 0e1ff06..32f1f76 100644
--- a/docs/0.5.0-beta/index.html
+++ b/docs/0.5.0-beta/index.html
@@ -3,7 +3,7 @@
<!DOCTYPE html>
<html lang="en">
<head>
- <meta name="generator" content="Hugo 0.58.3" />
+ <meta name="generator" content="Hugo 0.108.0">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
@@ -18,6 +18,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -202,9 +224,7 @@
</div>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -220,23 +240,16 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<h1 id="apache-hadoop-ozone">Apache Hadoop Ozone</h1>
-
-<figure>
- <img src="./ozone-usage.png" width="60%"/>
+<figure><img src="./ozone-usage.png" width="60%"/>
</figure>
-
<p><em><em>Ozone is a scalable, redundant, and distributed object store for Hadoop. <p>
Apart from scaling to billions of objects of varying sizes,
Ozone can function effectively in containerized environments
like Kubernetes.</em></em> <p></p>
-
<p>Applications like Apache Spark, Hive and YARN, work without any modifications when using Ozone. Ozone comes with a <a href="./interface/javaapi.html">Java client library</a>, <a href="./interface/s3.html">S3 protocol support</a>, and a <a href="./shell.html">command line interface</a> which makes it easy to use Ozone.</p>
-
<p>Ozone consists of volumes, buckets, and keys:</p>
-
<ul>
<li>Volumes are similar to user accounts. Only administrators can create or delete volumes.</li>
<li>Buckets are similar to directories. A bucket can contain any number of keys, but buckets cannot contain other buckets.</li>
diff --git a/docs/0.5.0-beta/index.xml b/docs/0.5.0-beta/index.xml
index 34a3cdf..5bea45d 100644
--- a/docs/0.5.0-beta/index.xml
+++ b/docs/0.5.0-beta/index.xml
@@ -1,16 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Overview on Ozone</title>
<link>/</link>
<description>Recent content in Overview on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
- <lastBuildDate>Fri, 07 Jun 2019 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Overview</title>
<link>/concept/overview.html</link>
@@ -206,7 +201,7 @@
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/startfromdockerhub.html</guid>
- <description>Requirements Working docker setup AWS CLI (optional) Ozone in a Single Container The easiest way to start up an all-in-one ozone container is to use the latest docker image from docker hub:
+ <description>Requirements Working docker setup AWS CLI (optional) Ozone in a Single Container The easiest way to start up an all-in-one ozone container is to use the latest docker image from docker hub:
docker run -p 9878:9878 -p 9876:9876 apache/ozone This command will pull down the ozone image from docker hub and start all ozone services in a single container. This container will run the required metadata servers (Ozone Manager, Storage Container Manager) one data node and the S3 compatible REST server (S3 Gateway).</description>
</item>
@@ -217,7 +212,7 @@ docker run -p 9878:9878 -p 9876:9876 apache/ozone This command will pull down th
<guid>/start/onprem.html</guid>
<description>If you are feeling adventurous, you can setup ozone in a real cluster. Setting up a real cluster requires us to understand the components of Ozone. Ozone is designed to work concurrently with HDFS. However, Ozone is also capable of running independently. The components of ozone are the same in both approaches.
-Ozone Components Ozone Manager - Is the server that is in charge of the namespace of Ozone.</description>
+Ozone Components Ozone Manager - Is the server that is in charge of the namespace of Ozone. Ozone Manager is responsible for all volume, bucket and key operations.</description>
</item>
<item>
@@ -226,10 +221,10 @@ Ozone Components Ozone Manager - Is the server that is in charge of the namespa
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/minikube.html</guid>
- <description>Requirements Working minikube setup kubectl kubernetes/examples folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. By default the kubernetes resource files are configured to use apache/ozone image from the dockerhub.
+ <description>Requirements Working minikube setup kubectl kubernetes/examples folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. By default the kubernetes resource files are configured to use apache/ozone image from the dockerhub.
To deploy it to minikube use the minikube configuration set:
-cd kubernetes/examples/minikube kubectl apply -f . And you can check the results with
-kubectl get pod Note: the kubernetes/examples/minikube resource set is optimized for minikube usage:</description>
+cd kubernetes/examples/minikube kubectl apply -f . And you can check the results with
+kubectl get pod Note: the kubernetes/examples/minikube resource set is optimized for minikube usage:</description>
</item>
<item>
@@ -238,7 +233,7 @@ kubectl get pod Note: the kubernetes/examples/minikube resource set is optimize
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/kubernetes.html</guid>
- <description>Requirements Working kubernetes cluster (LoadBalancer, PersistentVolume are not required) kubectl As the apache/ozone docker images are available from the dockerhub the deployment process is very similar to Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host) Deploy to kubernetes
+ <description>Requirements Working kubernetes cluster (LoadBalancer, PersistentVolume are not required) kubectl As the apache/ozone docker images are available from the dockerhub the deployment process is very similar to Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host) Deploy to kubernetes
kubernetes/examples folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases.</description>
</item>
@@ -248,9 +243,9 @@ kubernetes/examples folder of the ozone distribution contains kubernetes deploym
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/runningviadocker.html</guid>
- <description>Requirements docker and docker-compose Download the Ozone binary tarball and untar it.
- Go to the directory where the docker compose files exist and tell docker-compose to start Ozone in the background. This will start a small ozone instance on your machine.
- cd compose/ozone/ docker-compose up -d To verify that ozone is working as expected, let us log into a data node and run freon, the load generator for Ozone.</description>
+ <description>Requirements docker and docker-compose Download the Ozone binary tarball and untar it.
+Go to the directory where the docker compose files exist and tell docker-compose to start Ozone in the background. This will start a small ozone instance on your machine.
+cd compose/ozone/ docker-compose up -d To verify that ozone is working as expected, let us log into a data node and run freon, the load generator for Ozone. The exec datanode bash command will open a bash shell on the datanode.</description>
</item>
<item>
@@ -259,7 +254,7 @@ kubernetes/examples folder of the ozone distribution contains kubernetes deploym
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/fromsource.html</guid>
- <description>Requirements Java 1.8 Maven Protoc (2.5) This is a guide on how to build the ozone sources. If you are not planning to build sources yourself, you can safely skip this page. If you are a Hadoop ninja, and wise in the ways of Apache, you already know that a real Apache release is a source release.
+ <description>Requirements Java 1.8 Maven Protoc (2.5) This is a guide on how to build the ozone sources. If you are not planning to build sources yourself, you can safely skip this page. If you are a Hadoop ninja, and wise in the ways of Apache, you already know that a real Apache release is a source release.
If you want to build from sources, Please untar the source tarball and run the ozone build command.</description>
</item>
@@ -305,8 +300,7 @@ If you want to build from sources, Please untar the source tarball and run the o
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/design/decommissioning.html</guid>
- <description>title: Decommissioning in Ozone summary: Formal process to shut down machines in a safe way after the required replications. date: 2019-07-31 jira: HDDS-1881 status: current
-author: Anu Engineer, Marton Elek, Stephen O&rsquo;Donnell Abstract The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.
+ <description>title: Decommissioning in Ozone summary: Formal process to shut down machines in a safe way after the required replications. date: 2019-07-31 jira: HDDS-1881 status: current author: Anu Engineer, Marton Elek, Stephen O&rsquo;Donnell Abstract The goal of decommissioning is to turn off a selected set of machines without data loss. It may or may not require to move the existing replicas of the containers to other nodes.
There are two main classes of the decommissioning:</description>
</item>
@@ -338,4 +332,4 @@ There are two main classes of the decommissioning:</description>
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/interface.html b/docs/0.5.0-beta/interface.html
index b657e7d..ed60183 100644
--- a/docs/0.5.0-beta/interface.html
+++ b/docs/0.5.0-beta/interface.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -225,7 +247,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">Multi-Protocol Support </h3>
diff --git a/docs/0.5.0-beta/interface/index.xml b/docs/0.5.0-beta/interface/index.xml
index 1c4c332..3866714 100644
--- a/docs/0.5.0-beta/interface/index.xml
+++ b/docs/0.5.0-beta/interface/index.xml
@@ -1,16 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Programming Interfaces on Ozone</title>
<link>/interface.html</link>
<description>Recent content in Programming Interfaces on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
- <lastBuildDate>Thu, 14 Sep 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/interface/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/interface/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Java API</title>
<link>/interface/javaapi.html</link>
@@ -39,4 +34,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/interface/javaapi.html b/docs/0.5.0-beta/interface/javaapi.html
index ac99e38..3131322 100644
--- a/docs/0.5.0-beta/interface/javaapi.html
+++ b/docs/0.5.0-beta/interface/javaapi.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Java API</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,124 +255,99 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone ships with its own client library that supports RPC. For generic use cases the S3
compatible REST interface also can be used instead of the Ozone client.</p>
-
<h2 id="creating-an-ozone-client">Creating an Ozone client</h2>
-
<p>The Ozone client factory creates the ozone client. To get a RPC client we can call</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">OzoneClient <span style="color:#a6e22e">ozClient</span> <span style="color:#f92672">=</span> OzoneClientFactory.<span style="color:#a6e22e">getRpcClient</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>OzoneClient ozClient <span style="color:#f92672">=</span> OzoneClientFactory<span style="color:#f92672">.</span><span style="color:#a6e22e">getRpcClient</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>If the user want to create a client based on the configuration, then they can
call.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">OzoneClient <span style="color:#a6e22e">ozClient</span> <span style="color:#f92672">=</span> OzoneClientFactory.<span style="color:#a6e22e">getClient</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>OzoneClient ozClient <span style="color:#f92672">=</span> OzoneClientFactory<span style="color:#f92672">.</span><span style="color:#a6e22e">getClient</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>and an appropriate client based on configuration will be returned.</p>
-
<h2 id="writing-data-using-ozone-client">Writing data using Ozone Client</h2>
-
<p>The hierarchy of data inside ozone is a volume, bucket and a key. A volume
is a collection of buckets. A bucket is a collection of keys. To write data
to the ozone, you need a volume, bucket and a key.</p>
-
<h3 id="creating-a-volume">Creating a Volume</h3>
-
<p>Once we have a client, we need to get a reference to the ObjectStore. This
is done via</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">ObjectStore <span style="color:#a6e22e">objectStore</span> <span style="color:#f92672">=</span> ozClient.<span style="color:#a6e22e">getObjectStore</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>ObjectStore objectStore <span style="color:#f92672">=</span> ozClient<span style="color:#f92672">.</span><span style="color:#a6e22e">getObjectStore</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>An object store represents an active cluster against which the client is working.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// Let us create a volume to store our game assets.
-</span><span style="color:#75715e">// This uses default arguments for creating that volume.
-</span><span style="color:#75715e"></span>objectStore.<span style="color:#a6e22e">createVolume</span>(<span style="color:#e6db74">"assets"</span>);
-
-<span style="color:#75715e">// Let us verify that the volume got created.
-</span><span style="color:#75715e"></span>OzoneVolume <span style="color:#a6e22e">assets</span> <span style="color:#f92672">=</span> objectStore.<span style="color:#a6e22e">getVolume</span>(<span style="color:#e6db74">"assets"</span>);</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// Let us create a volume to store our game assets.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// This uses default arguments for creating that volume.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">createVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Let us verify that the volume got created.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneVolume assets <span style="color:#f92672">=</span> objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">getVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span></span></span></code></pre></div>
<p>It is possible to pass an array of arguments to the createVolume by creating volume arguments.</p>
-
<h3 id="creating-a-bucket">Creating a Bucket</h3>
-
<p>Once you have a volume, you can create buckets inside the volume.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// Let us create a bucket called videos.
-</span><span style="color:#75715e"></span>assets.<span style="color:#a6e22e">createBucket</span>(<span style="color:#e6db74">"videos"</span>);
-OzoneBucket <span style="color:#a6e22e">video</span> <span style="color:#f92672">=</span> assets.<span style="color:#a6e22e">getBucket</span>(<span style="color:#e6db74">"videos"</span>);</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// Let us create a bucket called videos.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>assets<span style="color:#f92672">.</span><span style="color:#a6e22e">createBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>OzoneBucket video <span style="color:#f92672">=</span> assets<span style="color:#f92672">.</span><span style="color:#a6e22e">getBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span></span></span></code></pre></div>
<p>At this point we have a usable volume and a bucket. Our volume is called <em>assets</em> and bucket is called <em>videos</em>.</p>
-
<p>Now we can create a Key.</p>
-
<h3 id="reading-and-writing-a-key">Reading and Writing a Key</h3>
-
<p>With a bucket object the users can now read and write keys. The following code reads a video called intro.mp4 from the local disk and stores in the <em>video</em> bucket that we just created.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// read data from the file, this is a user provided function.
-</span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> [] videoData <span style="color:#f92672">=</span> readFile(<span style="color:#e6db74">"intro.mp4"</span>);
-
-<span style="color:#75715e">// Create an output stream and write data.
-</span><span style="color:#75715e"></span>OzoneOutputStream <span style="color:#a6e22e">videoStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">createKey</span>(<span style="color:#e6db74">"intro.mp4"</span>, 1048576);
-videoStream.<span style="color:#a6e22e">write</span>(videoData);
-
-<span style="color:#75715e">// Close the stream when it is done.
-</span><span style="color:#75715e"></span>videoStream.<span style="color:#a6e22e">close</span>();
-
-
-<span style="color:#75715e">// We can use the same bucket to read the file that we just wrote, by creating an input Stream.
-</span><span style="color:#75715e">// Let us allocate a byte array to hold the video first.
-</span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span>[] <span style="color:#a6e22e">data</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span>[(<span style="color:#66d9ef">int</span>)1048576];
-OzoneInputStream <span style="color:#a6e22e">introStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">readKey</span>(<span style="color:#e6db74">"intro.mp4"</span>);
-<span style="color:#75715e">// read intro.mp4 into the data buffer
-</span><span style="color:#75715e"></span>introStream.<span style="color:#a6e22e">read</span>(data);
-introStream.<span style="color:#a6e22e">close</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// read data from the file, this is a user provided function.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> <span style="color:#f92672">[]</span> videoData <span style="color:#f92672">=</span> readFile<span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Create an output stream and write data.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneOutputStream videoStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">createKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">,</span> <span style="color:#ae81ff">1048576</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">write</span><span style="color:#f92672">(</span>videoData<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Close the stream when it is done.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// We can use the same bucket to read the file that we just wrote, by creating an input Stream.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// Let us allocate a byte array to hold the video first.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span><span style="color:#f92672">[]</span> data <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span><span style="color:#f92672">[(</span><span style="color:#66d9ef">int</span><span style="color:#f92672">)</span><span style="color:#ae81ff">1048576</span><span style="color:#f92672">];</span>
+</span></span><span style="display:flex;"><span>OzoneInputStream introStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">readKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// read intro.mp4 into the data buffer
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">read</span><span style="color:#f92672">(</span>data<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>Here is a complete example of the code that we just wrote. Please note the close functions being called in this program.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// Let us create a client
-</span><span style="color:#75715e"></span>OzoneClient <span style="color:#a6e22e">ozClient</span> <span style="color:#f92672">=</span> OzoneClientFactory.<span style="color:#a6e22e">getClient</span>();
-
-<span style="color:#75715e">// Get a reference to the ObjectStore using the client
-</span><span style="color:#75715e"></span>ObjectStore <span style="color:#a6e22e">objectStore</span> <span style="color:#f92672">=</span> ozClient.<span style="color:#a6e22e">getObjectStore</span>();
-
-<span style="color:#75715e">// Let us create a volume to store our game assets.
-</span><span style="color:#75715e">// This default arguments for creating that volume.
-</span><span style="color:#75715e"></span>objectStore.<span style="color:#a6e22e">createVolume</span>(<span style="color:#e6db74">"assets"</span>);
-
-<span style="color:#75715e">// Let us verify that the volume got created.
-</span><span style="color:#75715e"></span>OzoneVolume <span style="color:#a6e22e">assets</span> <span style="color:#f92672">=</span> objectStore.<span style="color:#a6e22e">getVolume</span>(<span style="color:#e6db74">"assets"</span>);
-
-<span style="color:#75715e">// Let us create a bucket called videos.
-</span><span style="color:#75715e"></span>assets.<span style="color:#a6e22e">createBucket</span>(<span style="color:#e6db74">"videos"</span>);
-OzoneBucket <span style="color:#a6e22e">video</span> <span style="color:#f92672">=</span> assets.<span style="color:#a6e22e">getBucket</span>(<span style="color:#e6db74">"videos"</span>);
-
-<span style="color:#75715e">// read data from the file, this is assumed to be a user provided function.
-</span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> [] videoData <span style="color:#f92672">=</span> readFile(<span style="color:#e6db74">"intro.mp4"</span>);
-
-<span style="color:#75715e">// Create an output stream and write data.
-</span><span style="color:#75715e"></span>OzoneOutputStream <span style="color:#a6e22e">videoStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">createKey</span>(<span style="color:#e6db74">"intro.mp4"</span>, 1048576);
-videoStream.<span style="color:#a6e22e">write</span>(videoData);
-
-<span style="color:#75715e">// Close the stream when it is done.
-</span><span style="color:#75715e"></span>videoStream.<span style="color:#a6e22e">close</span>();
-
-
-<span style="color:#75715e">// We can use the same bucket to read the file that we just wrote, by creating an input Stream.
-</span><span style="color:#75715e">// Let us allocate a byte array to hold the video first.
-</span><span style="color:#75715e"></span>
-<span style="color:#66d9ef">byte</span>[] <span style="color:#a6e22e">data</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span>[(<span style="color:#66d9ef">int</span>)1048576];
-OzoneInputStream <span style="color:#a6e22e">introStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">readKey</span>(<span style="color:#e6db74">"intro.mp4"</span>);
-introStream.<span style="color:#a6e22e">read</span>(data);
-
-<span style="color:#75715e">// Close the stream when it is done.
-</span><span style="color:#75715e"></span>introStream.<span style="color:#a6e22e">close</span>();
-
-<span style="color:#75715e">// Close the client.
-</span><span style="color:#75715e"></span>ozClient.<span style="color:#a6e22e">close</span>();</code></pre></div>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// Let us create a client
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneClient ozClient <span style="color:#f92672">=</span> OzoneClientFactory<span style="color:#f92672">.</span><span style="color:#a6e22e">getClient</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Get a reference to the ObjectStore using the client
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>ObjectStore objectStore <span style="color:#f92672">=</span> ozClient<span style="color:#f92672">.</span><span style="color:#a6e22e">getObjectStore</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Let us create a volume to store our game assets.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// This default arguments for creating that volume.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">createVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Let us verify that the volume got created.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneVolume assets <span style="color:#f92672">=</span> objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">getVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Let us create a bucket called videos.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>assets<span style="color:#f92672">.</span><span style="color:#a6e22e">createBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>OzoneBucket video <span style="color:#f92672">=</span> assets<span style="color:#f92672">.</span><span style="color:#a6e22e">getBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// read data from the file, this is assumed to be a user provided function.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> <span style="color:#f92672">[]</span> videoData <span style="color:#f92672">=</span> readFile<span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Create an output stream and write data.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneOutputStream videoStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">createKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">,</span> <span style="color:#ae81ff">1048576</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">write</span><span style="color:#f92672">(</span>videoData<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Close the stream when it is done.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// We can use the same bucket to read the file that we just wrote, by creating an input Stream.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// Let us allocate a byte array to hold the video first.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
+</span></span><span style="display:flex;"><span><span style="color:#66d9ef">byte</span><span style="color:#f92672">[]</span> data <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span><span style="color:#f92672">[(</span><span style="color:#66d9ef">int</span><span style="color:#f92672">)</span><span style="color:#ae81ff">1048576</span><span style="color:#f92672">];</span>
+</span></span><span style="display:flex;"><span>OzoneInputStream introStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">readKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">read</span><span style="color:#f92672">(</span>data<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Close the stream when it is done.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// Close the client.
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>ozClient<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span></span></span></code></pre></div>
diff --git a/docs/0.5.0-beta/interface/ozonefs.html b/docs/0.5.0-beta/interface/ozonefs.html
index 21f8181..b89d032 100644
--- a/docs/0.5.0-beta/interface/ozonefs.html
+++ b/docs/0.5.0-beta/interface/ozonefs.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Ozone File System</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,113 +255,75 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>The Hadoop compatible file system interface allows storage backends like Ozone
to be easily integrated into Hadoop eco-system. Ozone file system is an
Hadoop compatible file system.</p>
-
<h2 id="setting-up-the-ozone-file-system">Setting up the Ozone file system</h2>
-
<p>To create an ozone file system, we have to choose a bucket where the file system would live. This bucket will be used as the backend store for OzoneFileSystem. All the files and directories will be stored as keys in this bucket.</p>
-
<p>Please run the following commands to create a volume and bucket, if you don’t have them already.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume create /volume
-ozone sh bucket create /volume/bucket</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume create /volume
+</span></span><span style="display:flex;"><span>ozone sh bucket create /volume/bucket</span></span></code></pre></div>
<p>Once this is created, please make sure that bucket exists via the <em>list volume</em> or <em>list bucket</em> commands.</p>
-
<p>Please add the following entry to the core-site.xml.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.o3fs.impl<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzoneFileSystem<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span>
-<span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.AbstractFileSystem.o3fs.impl<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzFs<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span>
-<span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.defaultFS<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>o3fs://bucket.volume<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.o3fs.impl<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzoneFileSystem<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.AbstractFileSystem.o3fs.impl<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzFs<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.defaultFS<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>o3fs://bucket.volume<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span></span></span></code></pre></div>
<p>This will make this bucket to be the default file system for HDFS dfs commands and register the o3fs file system type.</p>
-
<p>You also need to add the ozone-filesystem.jar file to the classpath:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">export HADOOP_CLASSPATH<span style="color:#f92672">=</span>/opt/ozone/share/ozonefs/lib/hadoop-ozone-filesystem-lib-current*.jar:$HADOOP_CLASSPATH</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>export HADOOP_CLASSPATH<span style="color:#f92672">=</span>/opt/ozone/share/ozonefs/lib/hadoop-ozone-filesystem-lib-current*.jar:$HADOOP_CLASSPATH</span></span></code></pre></div>
<p>Once the default Filesystem has been setup, users can run commands like ls, put, mkdir, etc.
For example,</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls /</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls /</span></span></code></pre></div>
<p>or</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -mkdir /users</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -mkdir /users</span></span></code></pre></div>
<p>Or put command etc. In other words, all programs like Hive, Spark, and Distcp will work against this file system.
Please note that any keys created/deleted in the bucket using methods apart from OzoneFileSystem will show up as directories and files in the Ozone File System.</p>
-
<p>Note: Bucket and volume names are not allowed to have a period in them.
Moreover, the filesystem URI can take a fully qualified form with the OM host and an optional port as a part of the path following the volume name.
For example, you can specify both host and port:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:5678/key</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:5678/key</span></span></code></pre></div>
<p>When the port number is not specified, it will be retrieved from config key <code>ozone.om.address</code>
if defined; or it will fall back to the default port <code>9862</code>.
For example, we have <code>ozone.om.address</code> configured as following in <code>ozone-site.xml</code>:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.om.address<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>0.0.0.0:6789<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.om.address<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>0.0.0.0:6789<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span></span></span></code></pre></div>
<p>When we run command:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls o3fs://bucket.volume.om-host.example.com/key</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls o3fs://bucket.volume.om-host.example.com/key</span></span></code></pre></div>
<p>The above command is essentially equivalent to:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:6789/key</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:6789/key</span></span></code></pre></div>
<p>Note: Only port number from the config is used in this case,
whereas the host name in the config <code>ozone.om.address</code> is ignored.</p>
-
<h2 id="supporting-older-hadoop-version-legacy-jar-basicozonefilesystem">Supporting older Hadoop version (Legacy jar, BasicOzoneFilesystem)</h2>
-
<p>There are two ozonefs files, both of them include all the dependencies:</p>
-
<ul>
<li>share/ozone/lib/hadoop-ozone-filesystem-lib-current-VERSION.jar</li>
<li>share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-VERSION.jar</li>
</ul>
-
<p>The first one contains all the required dependency to use ozonefs with a
- compatible hadoop version (hadoop 3.2).</p>
-
+compatible hadoop version (hadoop 3.2).</p>
<p>The second one contains all the dependency in an internal, separated directory,
- and a special class loader is used to load all the classes from the location.</p>
-
+and a special class loader is used to load all the classes from the location.</p>
<p>With this method the hadoop-ozone-filesystem-lib-legacy.jar can be used from
- any older hadoop version (eg. hadoop 3.1, hadoop 2.7 or spark+hadoop 2.7)</p>
-
+any older hadoop version (eg. hadoop 3.1, hadoop 2.7 or spark+hadoop 2.7)</p>
<p>Similar to the dependency jar, there are two OzoneFileSystem implementation.</p>
-
<p>For hadoop 3.0 and newer, you can use <code>org.apache.hadoop.fs.ozone.OzoneFileSystem</code>
- which is a full implementation of the Hadoop compatible File System API.</p>
-
+which is a full implementation of the Hadoop compatible File System API.</p>
<p>For Hadoop 2.x you should use the Basic version: <code>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</code>.</p>
-
<p>This is the same implementation but doesn’t include the features/dependencies which are added with
- Hadoop 3.0. (eg. FS statistics, encryption zones).</p>
-
+Hadoop 3.0. (eg. FS statistics, encryption zones).</p>
<h3 id="summary">Summary</h3>
-
<p>The following table summarize which jar files and implementation should be used:</p>
-
<table>
<thead>
<tr>
@@ -350,37 +332,40 @@ whereas the host name in the config <code>ozone.om.address</code> is ignored.</p
<th>OzoneFileSystem implementation</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>3.2</td>
<td>filesystem-lib-current</td>
<td>org.apache.hadoop.fs.ozone.OzoneFileSystem</td>
</tr>
-
<tr>
<td>3.1</td>
<td>filesystem-lib-legacy</td>
<td>org.apache.hadoop.fs.ozone.OzoneFileSystem</td>
</tr>
-
<tr>
<td>2.9</td>
<td>filesystem-lib-legacy</td>
<td>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</td>
</tr>
-
<tr>
<td>2.7</td>
<td>filesystem-lib-legacy</td>
<td>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</td>
</tr>
+<tr>
+<td>With this method the hadoop-ozone-filesystem-lib-legacy.jar can be used from</td>
+<td></td>
+<td></td>
+</tr>
+<tr>
+<td>any older hadoop version (eg. hadoop 2.7 or spark+hadoop 2.7)</td>
+<td></td>
+<td></td>
+</tr>
</tbody>
</table>
-<p>With this method the hadoop-ozone-filesystem-lib-legacy.jar can be used from
- any older hadoop version (eg. hadoop 2.7 or spark+hadoop 2.7)</p>
-
<a class="btn btn-success btn-lg" href="../interface/s3.html">Next >></a>
diff --git a/docs/0.5.0-beta/interface/s3.html b/docs/0.5.0-beta/interface/s3.html
index 3339cd5..573b932 100644
--- a/docs/0.5.0-beta/interface/s3.html
+++ b/docs/0.5.0-beta/interface/s3.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>S3 Protocol</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,38 +255,25 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools.</p>
-
<h2 id="getting-started">Getting started</h2>
-
<p>S3 Gateway is a separated component which provides the S3 compatible APIs. It should be started additional to the regular Ozone components.</p>
-
<p>You can start a docker based cluster, including the S3 gateway from the release package.</p>
-
<p>Go to the <code>compose/ozones3</code> directory, and start the server:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose up -d</code></pre></div>
-<p>You can access the S3 gateway at <code>http://localhost:9878</code></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose up -d
+</span></span></code></pre></div><p>You can access the S3 gateway at <code>http://localhost:9878</code></p>
<h2 id="url-schema">URL Schema</h2>
-
-<p>Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. <a href="http://bucketname.host:9878">http://bucketname.host:9878</a>) and the path-style addresses (eg. <a href="http://host:9878/bucketname">http://host:9878/bucketname</a>)</p>
-
+<p>Ozone S3 gateway supports both the virtual-host-style URL s3 bucket addresses (eg. <a href="http://bucketname.host:9878">http://bucketname.host:9878</a>) and the path-style addresses (eg. http://host:9878/bucketname)</p>
<p>By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your <code>ozone-site.xml</code>:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.s3g.domain.name<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>s3g.internal<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span></code></pre></div>
-<h2 id="bucket-browser">Bucket browser</h2>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.s3g.domain.name<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>s3g.internal<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span>
+</span></span></code></pre></div><h2 id="bucket-browser">Bucket browser</h2>
<p>Buckets could be browsed from the browser by adding <code>?browser=true</code> to the bucket URL.</p>
-
-<p>For example the content of the ‘testbucket’ could be checked from the browser using the URL <a href="http://localhost:9878/testbucket?browser=true">http://localhost:9878/testbucket?browser=true</a></p>
-
+<p>For example the content of the ’testbucket’ could be checked from the browser using the URL http://localhost:9878/testbucket?browser=true</p>
<h2 id="implemented-rest-endpoints">Implemented REST endpoints</h2>
-
<p>Operations on S3Gateway service:</p>
-
<table>
<thead>
<tr>
@@ -274,7 +281,6 @@
<th>Status</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>GET service</td>
@@ -282,9 +288,7 @@
</tr>
</tbody>
</table>
-
<p>Operations on Bucket:</p>
-
<table>
<thead>
<tr>
@@ -293,32 +297,27 @@
<th>Notes</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>GET Bucket (List Objects) Version 2</td>
<td>implemented</td>
<td></td>
</tr>
-
<tr>
<td>HEAD Bucket</td>
<td>implemented</td>
<td></td>
</tr>
-
<tr>
<td>DELETE Bucket</td>
<td>implemented</td>
<td></td>
</tr>
-
<tr>
<td>PUT Bucket (Create bucket)</td>
<td>implemented</td>
<td></td>
</tr>
-
<tr>
<td>Delete Multiple Objects (POST)</td>
<td>implemented</td>
@@ -326,9 +325,7 @@
</tr>
</tbody>
</table>
-
<p>Operation on Objects:</p>
-
<table>
<thead>
<tr>
@@ -337,32 +334,27 @@
<th>Notes</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>PUT Object</td>
<td>implemented</td>
<td></td>
</tr>
-
<tr>
<td>GET Object</td>
<td>implemented</td>
<td></td>
</tr>
-
<tr>
<td>Multipart Upload</td>
<td>implemented</td>
<td>Except the listing of the current MultiPartUploads.</td>
</tr>
-
<tr>
<td>DELETE Object</td>
<td>implemented</td>
<td></td>
</tr>
-
<tr>
<td>HEAD Object</td>
<td>implemented</td>
@@ -370,42 +362,35 @@
</tr>
</tbody>
</table>
-
<h2 id="security">Security</h2>
-
<p>If security is not enabled, you can <em>use</em> <strong>any</strong> AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY</p>
-
<p>If security is enabled, you can get the key and the secret with the <code>ozone s3 getsecret</code> command (*kerberos based authentication is required).</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">/etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
-ozone s3 getsecret
-awsAccessKey<span style="color:#f92672">=</span>testuser/scm@EXAMPLE.COM
-awsSecret<span style="color:#f92672">=</span>c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999</code></pre></div>
-<p>Now, you can use the key and the secret to access the S3 endpoint:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">export AWS_ACCESS_KEY_ID<span style="color:#f92672">=</span>testuser/scm@EXAMPLE.COM
-export AWS_SECRET_ACCESS_KEY<span style="color:#f92672">=</span>c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
-aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1</code></pre></div>
-<h2 id="s3-bucket-name-mapping-to-ozone-buckets">S3 bucket name mapping to Ozone buckets</h2>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>/etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
+</span></span><span style="display:flex;"><span>ozone s3 getsecret
+</span></span><span style="display:flex;"><span>awsAccessKey<span style="color:#f92672">=</span>testuser/scm@EXAMPLE.COM
+</span></span><span style="display:flex;"><span>awsSecret<span style="color:#f92672">=</span>c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
+</span></span></code></pre></div><p>Now, you can use the key and the secret to access the S3 endpoint:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>export AWS_ACCESS_KEY_ID<span style="color:#f92672">=</span>testuser/scm@EXAMPLE.COM
+</span></span><span style="display:flex;"><span>export AWS_SECRET_ACCESS_KEY<span style="color:#f92672">=</span>c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
+</span></span><span style="display:flex;"><span>aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1
+</span></span></code></pre></div><h2 id="s3-bucket-name-mapping-to-ozone-buckets">S3 bucket name mapping to Ozone buckets</h2>
<p><strong>Note</strong>: Ozone has a notion for ‘volumes’ which is missing from the S3 Rest endpoint. Under the hood S3 bucket names are mapped to Ozone ‘volume/bucket’ locations (depending on the given authentication information).</p>
-
<p>To show the storage location of a S3 bucket, use the <code>ozone s3 path <bucketname></code> command.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket<span style="color:#f92672">=</span>bucket1
-
-ozone s3 path bucket1
-Volume name <span style="color:#66d9ef">for</span> S3Bucket is : s3thisisakey
-Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey</code></pre></div>
-<h2 id="clients">Clients</h2>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws s3api --endpoint-url http://localhost:9878 create-bucket --bucket<span style="color:#f92672">=</span>bucket1
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span>ozone s3 path bucket1
+</span></span><span style="display:flex;"><span>Volume name <span style="color:#66d9ef">for</span> S3Bucket is : s3thisisakey
+</span></span><span style="display:flex;"><span>Ozone FileSystem Uri is : o3fs://bucket1.s3thisisakey
+</span></span></code></pre></div><h2 id="clients">Clients</h2>
<h3 id="aws-cli">AWS Cli</h3>
-
<p><code>aws</code> CLI could be used by specifying the custom REST endpoint.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest</code></pre></div>
-<p>Or</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3 ls --endpoint http://localhost:9878 s3://buckettest</code></pre></div>
-<h3 id="s3-fuse-driver-goofys">S3 Fuse driver (goofys)</h3>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest
+</span></span></code></pre></div><p>Or</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws s3 ls --endpoint http://localhost:9878 s3://buckettest
+</span></span></code></pre></div><h3 id="s3-fuse-driver-goofys">S3 Fuse driver (goofys)</h3>
<p>Goofys is a S3 FUSE driver. It could be used to mount any Ozone bucket as posix file system.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1</code></pre></div>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>goofys --endpoint http://localhost:9878 bucket1 /mount/bucket1
+</span></span></code></pre></div>
</div>
diff --git a/docs/0.5.0-beta/recipe.html b/docs/0.5.0-beta/recipe.html
index 9aa85a2..f2ddb24 100644
--- a/docs/0.5.0-beta/recipe.html
+++ b/docs/0.5.0-beta/recipe.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -221,7 +243,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">Recipes of Ozone </h3>
diff --git a/docs/0.5.0-beta/recipe/index.xml b/docs/0.5.0-beta/recipe/index.xml
index 248f2c6..02d6854 100644
--- a/docs/0.5.0-beta/recipe/index.xml
+++ b/docs/0.5.0-beta/recipe/index.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Recipes on Ozone</title>
@@ -6,11 +6,7 @@
<description>Recent content in Recipes on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
- <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/recipe/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate><atom:link href="/recipe/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Monitoring with Prometheus</title>
<link>/recipe/prometheus.html</link>
@@ -30,4 +26,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/recipe/prometheus.html b/docs/0.5.0-beta/recipe/prometheus.html
index a32e29a..d240f9b 100644
--- a/docs/0.5.0-beta/recipe/prometheus.html
+++ b/docs/0.5.0-beta/recipe/prometheus.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Monitoring with Prometheus</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,77 +251,68 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p><a href="https://prometheus.io/">Prometheus</a> is an open-source monitoring server developed under under the <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a>.</p>
-
<p>Ozone supports Prometheus out of the box. The servers start a prometheus
compatible metrics endpoint where all the available hadoop metrics are published in prometheus exporter format.</p>
-
<h2 id="prerequisites">Prerequisites</h2>
-
<ol>
<li><a href="../start/runningviadocker.html">Install the and start</a> an Ozone cluster.</li>
<li><a href="https://prometheus.io/download/#prometheus">Download</a> the prometheus binary.</li>
</ol>
-
<h2 id="monitoring-with-prometheus">Monitoring with prometheus</h2>
-
<ul>
-<li><p>To enable the Prometheus metrics endpoint you need to add a new configuration to the <code>ozone-site.xml</code> file.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><property></span>
-<span style="color:#f92672"><name></span>hdds.prometheus.endpoint.enabled<span style="color:#f92672"></name></span>
-<span style="color:#f92672"><value></span>true<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span></code></pre></div></li>
+<li>To enable the Prometheus metrics endpoint you need to add a new configuration to the <code>ozone-site.xml</code> file.</li>
</ul>
-
-<p><em>Note</em>: for Docker compose based pseudo cluster put the <br />
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>hdds.prometheus.endpoint.enabled<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>true<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span>
+</span></span></code></pre></div><p><em>Note</em>: for Docker compose based pseudo cluster put the <br>
<code>OZONE-SITE.XML_hdds.prometheus.endpoint.enabled=true</code> line to the <code>docker-config</code> file.</p>
-
<ul>
-<li><p>Restart the Ozone Manager and Storage Container Manager and check the prometheus endpoints:</p>
-
-<ul>
-<li><p><a href="http://scm:9874/prom">http://scm:9874/prom</a></p></li>
-
-<li><p><a href="http://ozoneManager:9876/prom">http://ozoneManager:9876/prom</a></p></li>
-</ul></li>
-
-<li><p>Create a prometheus.yaml configuration with the previous endpoints:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">global:
-scrape_interval: 15s
-
-scrape_configs:
-- job_name: ozone
-metrics_path: /prom
-static_configs:
- - targets:
- - <span style="color:#e6db74">"scm:9876"</span>
- - <span style="color:#e6db74">"ozoneManager:9874"</span></code></pre></div></li>
-
-<li><p>Start with prometheus from the directory where you have the prometheus.yaml file:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">prometheus</code></pre></div></li>
-
-<li><p>Check the active targets in the prometheus web-ui:</p></li>
+<li>
+<p>Restart the Ozone Manager and Storage Container Manager and check the prometheus endpoints:</p>
+</li>
+<li>
+<p>http://scm:9874/prom</p>
+</li>
+<li>
+<p>http://ozoneManager:9876/prom</p>
+</li>
+<li>
+<p>Create a prometheus.yaml configuration with the previous endpoints:</p>
+</li>
</ul>
-
-<p><a href="http://localhost:9090/targets">http://localhost:9090/targets</a></p>
-
-<p><img src="prometheus.png" alt="Prometheus target page example" /></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">global</span>:
+</span></span><span style="display:flex;"><span> <span style="color:#f92672">scrape_interval</span>: <span style="color:#ae81ff">15s</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">scrape_configs</span>:
+</span></span><span style="display:flex;"><span> - <span style="color:#f92672">job_name</span>: <span style="color:#ae81ff">ozone</span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672">metrics_path</span>: <span style="color:#ae81ff">/prom</span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672">static_configs</span>:
+</span></span><span style="display:flex;"><span> - <span style="color:#f92672">targets</span>:
+</span></span><span style="display:flex;"><span> - <span style="color:#e6db74">"scm:9876"</span>
+</span></span><span style="display:flex;"><span> - <span style="color:#e6db74">"ozoneManager:9874"</span>
+</span></span></code></pre></div><ul>
+<li>Start with prometheus from the directory where you have the prometheus.yaml file:</li>
+</ul>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>prometheus
+</span></span></code></pre></div><ul>
+<li>Check the active targets in the prometheus web-ui:</li>
+</ul>
+<p>http://localhost:9090/targets</p>
+<p><img src="prometheus.png" alt="Prometheus target page example"></p>
<ul>
-<li>Check any metrics on the prometheus web ui.<br />
+<li>Check any metrics on the prometheus web ui.<br>
For example:</li>
</ul>
-
-<p><a href="http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1">http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1</a></p>
-
-<p><img src="prometheus-key-allocate.png" alt="Prometheus metrics page example" /></p>
-
+<p>http://localhost:9090/graph?g0.range_input=1h&g0.expr=om_metrics_num_key_allocate&g0.tab=1</p>
+<p><img src="prometheus-key-allocate.png" alt="Prometheus metrics page example"></p>
<h2 id="note">Note</h2>
-
<p>The ozone distribution contains a ready-to-use, dockerized environment to try out ozone and prometheus. It can be found under <code>compose/ozoneperf</code> directory.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd compose/ozoneperf
-docker-compose up -d</code></pre></div>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd compose/ozoneperf
+</span></span><span style="display:flex;"><span>docker-compose up -d
+</span></span></code></pre></div>
<a class="btn btn-success btn-lg" href="../recipe/sparkozonefsk8s.html">Next >></a>
diff --git a/docs/0.5.0-beta/recipe/sparkozonefsk8s.html b/docs/0.5.0-beta/recipe/sparkozonefsk8s.html
index 34bde22..9213bee 100644
--- a/docs/0.5.0-beta/recipe/sparkozonefsk8s.html
+++ b/docs/0.5.0-beta/recipe/sparkozonefsk8s.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Spark in Kubernetes with OzoneFS</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,133 +251,103 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>This recipe shows how Ozone object store can be used from Spark using:</p>
-
<ul>
<li>OzoneFS (Hadoop compatible file system)</li>
<li>Hadoop 2.7 (included in the Spark distribution)</li>
<li>Kubernetes Spark scheduler</li>
<li>Local spark client</li>
</ul>
-
<h2 id="requirements">Requirements</h2>
-
<p>Download latest Spark and Ozone distribution and extract them. This method is
tested with the <code>spark-2.4.0-bin-hadoop2.7</code> distribution.</p>
-
<p>You also need the following:</p>
-
<ul>
<li>A container repository to push and pull the spark+ozone images. (In this recipe we will use the dockerhub)</li>
<li>A repo/name for the custom containers (in this recipe <em>myrepo/ozone-spark</em>)</li>
<li>A dedicated namespace in kubernetes (we use <em>yournamespace</em> in this recipe)</li>
</ul>
-
<h2 id="create-the-docker-image-for-drivers">Create the docker image for drivers</h2>
-
-<h3 id="create-the-base-spark-driver-executor-image">Create the base Spark driver/executor image</h3>
-
+<h3 id="create-the-base-spark-driverexecutor-image">Create the base Spark driver/executor image</h3>
<p>First of all create a docker image with the Spark image creator.
Execute the following from the Spark distribution</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">./bin/docker-image-tool.sh -r myrepo -t <span style="color:#ae81ff">2</span>.4.0 build</code></pre></div>
-<p><em>Note</em>: if you use Minikube add the <code>-m</code> flag to use the docker daemon of the Minikube image:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">./bin/docker-image-tool.sh -m -r myrepo -t <span style="color:#ae81ff">2</span>.4.0 build</code></pre></div>
-<p><code>./bin/docker-image-tool.sh</code> is an official Spark tool to create container images and this step will create multiple Spark container images with the name <em>myrepo/spark</em>. The first container will be used as a base container in the following steps.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>./bin/docker-image-tool.sh -r myrepo -t 2.4.0 build
+</span></span></code></pre></div><p><em>Note</em>: if you use Minikube add the <code>-m</code> flag to use the docker daemon of the Minikube image:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>./bin/docker-image-tool.sh -m -r myrepo -t 2.4.0 build
+</span></span></code></pre></div><p><code>./bin/docker-image-tool.sh</code> is an official Spark tool to create container images and this step will create multiple Spark container images with the name <em>myrepo/spark</em>. The first container will be used as a base container in the following steps.</p>
<h3 id="customize-the-docker-image">Customize the docker image</h3>
-
<p>Create a new directory for customizing the created docker image.</p>
-
<p>Copy the <code>ozone-site.xml</code> from the cluster:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">kubectl cp om-0:/opt/hadoop/etc/hadoop/ozone-site.xml .</code></pre></div>
-<p>And create a custom <code>core-site.xml</code>.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><configuration></span>
- <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.o3fs.impl<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span>
- <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.AbstractFileSystem.o3fs.impl<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzFs<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span>
-<span style="color:#f92672"></configuration></span></code></pre></div>
-<p><em>Note</em>: You may also use <code>org.apache.hadoop.fs.ozone.OzoneFileSystem</code> without the <code>Basic</code> prefix. The <code>Basic</code> version doesn’t support FS statistics and encryption zones but can work together with older hadoop versions.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl cp om-0:/opt/hadoop/etc/hadoop/ozone-site.xml .
+</span></span></code></pre></div><p>And create a custom <code>core-site.xml</code>.</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#f92672"><configuration></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.o3fs.impl<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.AbstractFileSystem.o3fs.impl<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzFs<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></configuration></span>
+</span></span></code></pre></div><p><em>Note</em>: You may also use <code>org.apache.hadoop.fs.ozone.OzoneFileSystem</code> without the <code>Basic</code> prefix. The <code>Basic</code> version doesn’t support FS statistics and encryption zones but can work together with older hadoop versions.</p>
<p>Copy the <code>ozonefs.jar</code> file from an ozone distribution (<strong>use the legacy version!</strong>)</p>
-
-<pre><code>kubectl cp om-0:/opt/hadoop/share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-0.4.0-SNAPSHOT.jar .
-</code></pre>
-
-<p>Create a new Dockerfile and build the image:</p>
-
-<pre><code>FROM myrepo/spark:2.4.0
+<pre tabindex="0"><code>kubectl cp om-0:/opt/hadoop/share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-0.4.0-SNAPSHOT.jar .
+</code></pre><p>Create a new Dockerfile and build the image:</p>
+<pre tabindex="0"><code>FROM myrepo/spark:2.4.0
ADD core-site.xml /opt/hadoop/conf/core-site.xml
ADD ozone-site.xml /opt/hadoop/conf/ozone-site.xml
ENV HADOOP_CONF_DIR=/opt/hadoop/conf
ENV SPARK_EXTRA_CLASSPATH=/opt/hadoop/conf
ADD hadoop-ozone-filesystem-lib-legacy-0.4.0-SNAPSHOT.jar /opt/hadoop-ozone-filesystem-lib-legacy.jar
-</code></pre>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker build -t myrepo/spark-ozone</code></pre></div>
-<p>For remote kubernetes cluster you may need to push it:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker push myrepo/spark-ozone</code></pre></div>
-<h2 id="create-a-bucket-and-identify-the-ozonefs-path">Create a bucket and identify the ozonefs path</h2>
-
+</code></pre><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker build -t myrepo/spark-ozone
+</span></span></code></pre></div><p>For remote kubernetes cluster you may need to push it:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker push myrepo/spark-ozone
+</span></span></code></pre></div><h2 id="create-a-bucket-and-identify-the-ozonefs-path">Create a bucket and identify the ozonefs path</h2>
<p>Download any text file and put it to the <code>/tmp/alice.txt</code> first.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">kubectl port-forward s3g-0 <span style="color:#ae81ff">9878</span>:9878
-aws s3api --endpoint http://localhost:9878 create-bucket --bucket<span style="color:#f92672">=</span>test
-aws s3api --endpoint http://localhost:9878 put-object --bucket test --key alice.txt --body /tmp/alice.txt
-kubectl exec -it scm-0 ozone s3 path test</code></pre></div>
-<p>The output of the last command is something like this:</p>
-
-<pre><code>Volume name for S3Bucket is : s3asdlkjqiskjdsks
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl port-forward s3g-0 9878:9878
+</span></span><span style="display:flex;"><span>aws s3api --endpoint http://localhost:9878 create-bucket --bucket<span style="color:#f92672">=</span>test
+</span></span><span style="display:flex;"><span>aws s3api --endpoint http://localhost:9878 put-object --bucket test --key alice.txt --body /tmp/alice.txt
+</span></span><span style="display:flex;"><span>kubectl exec -it scm-0 ozone s3 path test
+</span></span></code></pre></div><p>The output of the last command is something like this:</p>
+<pre tabindex="0"><code>Volume name for S3Bucket is : s3asdlkjqiskjdsks
Ozone FileSystem Uri is : o3fs://test.s3asdlkjqiskjdsks
-</code></pre>
-
-<p>Write down the ozone filesystem uri as it should be used with the spark-submit command.</p>
-
+</code></pre><p>Write down the ozone filesystem uri as it should be used with the spark-submit command.</p>
<h2 id="create-service-account-to-use">Create service account to use</h2>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">kubectl create serviceaccount spark -n yournamespace
-kubectl create clusterrolebinding spark-role --clusterrole<span style="color:#f92672">=</span>edit --serviceaccount<span style="color:#f92672">=</span>yournamespace:spark --namespace<span style="color:#f92672">=</span>yournamespace</code></pre></div>
-<h2 id="execute-the-job">Execute the job</h2>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl create serviceaccount spark -n yournamespace
+</span></span><span style="display:flex;"><span>kubectl create clusterrolebinding spark-role --clusterrole<span style="color:#f92672">=</span>edit --serviceaccount<span style="color:#f92672">=</span>yournamespace:spark --namespace<span style="color:#f92672">=</span>yournamespace
+</span></span></code></pre></div><h2 id="execute-the-job">Execute the job</h2>
<p>Execute the following spark-submit command, but change at least the following values:</p>
-
<ul>
<li>the kubernetes master url (you can check your <em>~/.kube/config</em> to find the actual value)</li>
<li>the kubernetes namespace (<em>yournamespace</em> in this example)</li>
<li>serviceAccountName (you can use the <em>spark</em> value if you followed the previous steps)</li>
<li>container.image (in this example this is <em>myrepo/spark-ozone</em>. This is pushed to the registry in the previous steps)</li>
-
-<li><p>location of the input file (o3fs://…), use the string which is identified earlier with the <br />
-<code>ozone s3 path <bucketname></code> command</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">bin/spark-submit <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--master k8s://https://kubernetes:6443 <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--deploy-mode cluster <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--name spark-word-count <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--class org.apache.spark.examples.JavaWordCount <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--conf spark.executor.instances<span style="color:#f92672">=</span><span style="color:#ae81ff">1</span> <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--conf spark.kubernetes.namespace<span style="color:#f92672">=</span>yournamespace <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--conf spark.kubernetes.authenticate.driver.serviceAccountName<span style="color:#f92672">=</span>spark <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--conf spark.kubernetes.container.image<span style="color:#f92672">=</span>myrepo/spark-ozone <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--conf spark.kubernetes.container.image.pullPolicy<span style="color:#f92672">=</span>Always <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>--jars /opt/hadoop-ozone-filesystem-lib-legacy.jar <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar <span style="color:#ae81ff">\
-</span><span style="color:#ae81ff"></span>o3fs://bucket.volume/alice.txt</code></pre></div></li>
+<li>location of the input file (o3fs://…), use the string which is identified earlier with the <br>
+<code>ozone s3 path <bucketname></code> command</li>
</ul>
-
-<p>Check the available <code>spark-word-count-...</code> pods with <code>kubectl get pod</code></p>
-
-<p>Check the output of the calculation with <br />
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>bin/spark-submit <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --master k8s://https://kubernetes:6443 <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --deploy-mode cluster <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --name spark-word-count <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --class org.apache.spark.examples.JavaWordCount <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.executor.instances<span style="color:#f92672">=</span><span style="color:#ae81ff">1</span> <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.kubernetes.namespace<span style="color:#f92672">=</span>yournamespace <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.kubernetes.authenticate.driver.serviceAccountName<span style="color:#f92672">=</span>spark <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.kubernetes.container.image<span style="color:#f92672">=</span>myrepo/spark-ozone <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.kubernetes.container.image.pullPolicy<span style="color:#f92672">=</span>Always <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --jars /opt/hadoop-ozone-filesystem-lib-legacy.jar <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> o3fs://bucket.volume/alice.txt
+</span></span></code></pre></div><p>Check the available <code>spark-word-count-...</code> pods with <code>kubectl get pod</code></p>
+<p>Check the output of the calculation with <br>
<code>kubectl logs spark-word-count-1549973913699-driver</code></p>
-
<p>You should see the output of the wordcount job. For example:</p>
-
-<pre><code>...
+<pre tabindex="0"><code>...
name: 8
William: 3
-this,': 1
-SOUP!': 1
+this,': 1
+SOUP!': 1
`Silence: 1
`Mine: 1
ordered.: 1
@@ -367,7 +357,6 @@ candle: 2
...
</code></pre>
-
</div>
diff --git a/docs/0.5.0-beta/security.html b/docs/0.5.0-beta/security.html
index 8c90896..38c7458 100644
--- a/docs/0.5.0-beta/security.html
+++ b/docs/0.5.0-beta/security.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -221,7 +243,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">Securing Ozone </h3>
@@ -234,12 +255,10 @@
</p>
</div>
</div>
-
<div class="alert alert-warning" role="alert">
If you would like to understand Ozone's security architecture at a greater
depth, please take a look at <a href="https://issues.apache.org/jira/secure/attachment/12911638/HadoopStorageLayerSecurity.pdf">Ozone security architecture.</a>
</div>
-
<p>Depending on your needs, there are multiple optional steps in securing ozone.</p>
diff --git a/docs/0.5.0-beta/security/index.xml b/docs/0.5.0-beta/security/index.xml
index 479487a..569524c 100644
--- a/docs/0.5.0-beta/security/index.xml
+++ b/docs/0.5.0-beta/security/index.xml
@@ -1,15 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Security on Ozone</title>
<link>/security.html</link>
<description>Recent content in Security on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
-
- <atom:link href="/security/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/security/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Securing Ozone</title>
<link>/security/secureozone.html</link>
@@ -65,4 +61,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/security/secuitywithranger.html b/docs/0.5.0-beta/security/secuitywithranger.html
index 0b7ca3e..54575dd 100644
--- a/docs/0.5.0-beta/security/secuitywithranger.html
+++ b/docs/0.5.0-beta/security/secuitywithranger.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -229,20 +251,16 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Apache Ranger™ is a framework to enable, monitor and manage comprehensive data
security across the Hadoop platform. Any version of Apache Ranger which is greater
than 1.20 is aware of Ozone, and can manage an Ozone cluster.</p>
-
<p>To use Apache Ranger, you must have Apache Ranger installed in your Hadoop
Cluster. For installation instructions of Apache Ranger, Please take a look
at the <a href="https://ranger.apache.org/index.html">Apache Ranger website</a>.</p>
-
<p>If you have a working Apache Ranger installation that is aware of Ozone, then
configuring Ozone to work with Apache Ranger is trivial. You have to enable
the ACLs support and set the acl authorizer class inside Ozone to be Ranger
authorizer. Please add the following properties to the ozone-site.xml.</p>
-
<table>
<thead>
<tr>
@@ -250,13 +268,11 @@ authorizer. Please add the following properties to the ozone-site.xml.</p>
<th>Value</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>ozone.acl.enabled</td>
<td>true</td>
</tr>
-
<tr>
<td>ozone.acl.authorizer.class</td>
<td>org.apache.ranger.authorization.ozone.authorizer.RangerOzoneAuthorizer</td>
diff --git a/docs/0.5.0-beta/security/secureozone.html b/docs/0.5.0-beta/security/secureozone.html
index 6c837e4..6470810 100644
--- a/docs/0.5.0-beta/security/secureozone.html
+++ b/docs/0.5.0-beta/security/secureozone.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Securing Ozone</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,18 +251,14 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<h1 id="kerberos">Kerberos</h1>
-
<p>Ozone depends on <a href="https://web.mit.edu/kerberos/">Kerberos</a> to make the
clusters secure. Historically, HDFS has supported running in an isolated
secure networks where it is possible to deploy without securing the cluster.</p>
-
<p>This release of Ozone follows that model, but soon will move to <em>secure by
default.</em> Today to enable security in ozone cluster, we need to set the
configuration <strong>ozone.security.enabled</strong> to <em>true</em> and <strong>hadoop.security.authentication</strong>
to <em>kerberos</em>.</p>
-
<table>
<thead>
<tr>
@@ -250,55 +266,41 @@ to <em>kerberos</em>.</p>
<th>Value</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>ozone.security.enabled</td>
<td><em>true</em></td>
</tr>
-
<tr>
<td>hadoop.security.authentication</td>
<td><em>kerberos</em></td>
</tr>
</tbody>
</table>
-
<h1 id="tokens">Tokens</h1>
-
<p>Ozone uses a notion of tokens to avoid overburdening the Kerberos server.
When you serve thousands of requests per second, involving Kerberos might not
work well. Hence once an authentication is done, Ozone issues delegation
tokens and block tokens to the clients. These tokens allow applications to do
specified operations against the cluster, as if they have kerberos tickets
with them. Ozone supports following kinds of tokens.</p>
-
<h3 id="delegation-token">Delegation Token</h3>
-
<p>Delegation tokens allow an application to impersonate a users kerberos
credentials. This token is based on verification of kerberos identity and is
issued by the Ozone Manager. Delegation tokens are enabled by default when
security is enabled.</p>
-
<h3 id="block-token">Block Token</h3>
-
<p>Block tokens allow a client to read or write a block. This is needed so that
data nodes know that the user/client has permission to read or make
modifications to the block.</p>
-
<h3 id="s3authinfo">S3AuthInfo</h3>
-
<p>S3 uses a very different shared secret security scheme. Ozone supports the AWS Signature Version 4 protocol,
and from the end users perspective Ozone’s S3 feels exactly like AWS S3.</p>
-
<p>The S3 credential tokens are called S3 auth info in the code. These tokens are
also enabled by default when security is enabled.</p>
-
<p>Each of the service daemons that make up Ozone needs a Kerberos service
principal name and a corresponding <a href="https://web.mit.edu/kerberos/krb5-latest/doc/basic/keytab_def.html">kerberos key tab</a> file.</p>
-
<p>All these settings should be made in ozone-site.xml.</p>
-
<div class="card-group">
<div class="card">
<div class="card-body">
diff --git a/docs/0.5.0-beta/security/securingdatanodes.html b/docs/0.5.0-beta/security/securingdatanodes.html
index 57332bb..d7e04e4 100644
--- a/docs/0.5.0-beta/security/securingdatanodes.html
+++ b/docs/0.5.0-beta/security/securingdatanodes.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Securing Datanodes</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,16 +251,13 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Datanodes under Hadoop is traditionally secured by creating a Keytab file on
the data nodes. With Ozone, we have moved away to using data node
certificates. That is, Kerberos on data nodes is not needed in case of a
secure Ozone cluster.</p>
-
<p>However, we support the legacy Kerberos based Authentication to make it easy
for the current set of users.The HDFS configuration keys are the following
that is setup in hdfs-site.xml.</p>
-
<table>
<thead>
<tr>
@@ -248,54 +265,44 @@ that is setup in hdfs-site.xml.</p>
<th>Description</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>dfs.datanode.kerberos.principal</td>
-<td>The datanode service principal. <br/> e.g. dn/_HOST@REALM.COM</td>
+<td>The datanode service principal. <br/> e.g. <a href="mailto:dn/_HOST@REALM.COM">dn/_HOST@REALM.COM</a></td>
</tr>
-
<tr>
<td>dfs.datanode.keytab.file</td>
<td>The keytab file used by datanode daemon to login as its service principal.</td>
</tr>
-
<tr>
<td>hdds.datanode.http.kerberos.principal</td>
<td>Datanode http server service principal.</td>
</tr>
-
<tr>
<td>hdds.datanode.http.kerberos.keytab</td>
<td>The keytab file used by datanode http server to login as its service principal.</td>
</tr>
</tbody>
</table>
-
<h2 id="how-a-data-node-becomes-secure">How a data node becomes secure.</h2>
-
<p>Under Ozone, when a data node boots up and discovers SCM’s address, the first
thing that data node does is to create a private key and send a certificate
request to the SCM.</p>
-
-<p><h3>Certificate Approval via Kerberos <span class="badge badge-secondary">Current Model</span></h3>
+<h3>Certificate Approval via Kerberos <span class="badge badge-secondary">Current Model</span></h3>
SCM has a built-in CA, and SCM has to approve this request. If the data node
already has a Kerberos key tab, then SCM will trust Kerberos credentials and
-issue a certificate automatically.</p>
-
-<p><h3>Manual Approval <span class="badge badge-primary">In Progress</span></h3>
+issue a certificate automatically.
+<h3>Manual Approval <span class="badge badge-primary">In Progress</span></h3>
If these are band new data nodes and Kerberos key tabs are not present at the
data nodes, then this request for the data nodes identity certificate is
queued up for approval from the administrator(This is work in progress,
not committed in Ozone yet). In other words, the web of trust is established
-by the administrator of the cluster.</p>
-
-<p><h3>Automatic Approval <span class="badge badge-secondary">In Progress</span></h3>
+by the administrator of the cluster.
+<h3>Automatic Approval <span class="badge badge-secondary">In Progress</span></h3>
If you running under an container orchestrator like Kubernetes, we rely on
Kubernetes to create a one-time token that will be given to data node during
boot time to prove the identity of the data node container (This is also work
-in progress.)</p>
-
+in progress.)
<p>Once a certificate is issued, a data node is secure and Ozone manager can
issue block tokens. If there is no data node certificates or the SCM’s root
certificate is not present in the data node, then data node will register
diff --git a/docs/0.5.0-beta/security/securings3.html b/docs/0.5.0-beta/security/securings3.html
index 64e7555..7c3a0e4 100644
--- a/docs/0.5.0-beta/security/securings3.html
+++ b/docs/0.5.0-beta/security/securings3.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -229,41 +251,34 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>To access an S3 bucket, users need AWS access key ID and AWS secret. Both of
these are generated by going to AWS website. When you use Ozone’s S3
protocol, you need the same AWS access key and secret.</p>
-
<p>Under Ozone, the clients can download the access key directly from Ozone.
The user needs to <code>kinit</code> first and once they have authenticated via kerberos
- they can download the S3 access key ID and AWS secret. Just like AWS S3,
- both of these are secrets that needs to be protected by the client since it
- gives full access to the S3 buckets.</p>
-
+they can download the S3 access key ID and AWS secret. Just like AWS S3,
+both of these are secrets that needs to be protected by the client since it
+gives full access to the S3 buckets.</p>
<ul>
-<li><p>S3 clients can get the secret access id and user secret from OzoneManager.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone s3 getsecret</code></pre></div></li>
+<li>S3 clients can get the secret access id and user secret from OzoneManager.</li>
</ul>
-
-<p>This command will talk to ozone, validate the user via kerberos and generate
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone s3 getsecret
+</span></span></code></pre></div><p>This command will talk to ozone, validate the user via kerberos and generate
the AWS credentials. The values will be printed out on the screen. You can
set these values up in your <em>.aws</em> file for automatic access while working
against Ozone S3 buckets.</p>
-
<div class="alert alert-danger" role="alert">
Please note: These S3 crediantials are like your kerberos passswords
that give compelete access to your buckets.
</div>
-
<ul>
-<li><p>Now you can proceed to setup these secrets in aws configs:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws configure set default.s3.signature_version s3v4
-aws configure set aws_access_key_id <span style="color:#e6db74">${</span>accessId<span style="color:#e6db74">}</span>
-aws configure set aws_secret_access_key <span style="color:#e6db74">${</span>secret<span style="color:#e6db74">}</span>
-aws configure set region us-west-1</code></pre></div></li>
+<li>Now you can proceed to setup these secrets in aws configs:</li>
</ul>
-
-<p>Please refer to AWS S3 documentation on how to use S3 via command line or via
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws configure set default.s3.signature_version s3v4
+</span></span><span style="display:flex;"><span>aws configure set aws_access_key_id <span style="color:#e6db74">${</span>accessId<span style="color:#e6db74">}</span>
+</span></span><span style="display:flex;"><span>aws configure set aws_secret_access_key <span style="color:#e6db74">${</span>secret<span style="color:#e6db74">}</span>
+</span></span><span style="display:flex;"><span>aws configure set region us-west-1
+</span></span></code></pre></div><p>Please refer to AWS S3 documentation on how to use S3 via command line or via
S3 API.</p>
diff --git a/docs/0.5.0-beta/security/securingtde.html b/docs/0.5.0-beta/security/securingtde.html
index 63b358a..8221e5e 100644
--- a/docs/0.5.0-beta/security/securingtde.html
+++ b/docs/0.5.0-beta/security/securingtde.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Transparent Data Encryption</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,17 +251,13 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone TDE setup process and usage are very similar to HDFS TDE.
The major difference is that Ozone TDE is enabled at Ozone bucket level
when a bucket is created.</p>
-
<h3 id="setting-up-the-key-management-server">Setting up the Key Management Server</h3>
-
<p>To use TDE, clients must setup a Key Management Server and provide that URI to
Ozone/HDFS. Since Ozone and HDFS can use the same Key Management Server, this
- configuration can be provided via <em>hdfs-site.xml</em>.</p>
-
+configuration can be provided via <em>hdfs-site.xml</em>.</p>
<table>
<thead>
<tr>
@@ -249,7 +265,6 @@ Ozone/HDFS. Since Ozone and HDFS can use the same Key Management Server, this
<th>Value</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>hadoop.security.key.provider.path</td>
@@ -257,30 +272,23 @@ Ozone/HDFS. Since Ozone and HDFS can use the same Key Management Server, this
</tr>
</tbody>
</table>
-
<h3 id="using-transparent-data-encryption">Using Transparent Data Encryption</h3>
-
<p>If this is already configured for your cluster, then you can simply proceed
to create the encryption key and enable encrypted buckets.</p>
-
<p>To create an encrypted bucket, client need to:</p>
-
<ul>
-<li><p>Create a bucket encryption key with hadoop key CLI, which is similar to
-how you would use HDFS encryption zones.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hadoop key create encKey</code></pre></div></li>
+<li>Create a bucket encryption key with hadoop key CLI, which is similar to
+how you would use HDFS encryption zones.</li>
</ul>
-
-<p>The above command creates an encryption key for the bucket you want to protect.
- Once the key is created, you can tell Ozone to use that key when you are
- reading and writing data into a bucket.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hadoop key create encKey
+</span></span></code></pre></div><p>The above command creates an encryption key for the bucket you want to protect.
+Once the key is created, you can tell Ozone to use that key when you are
+reading and writing data into a bucket.</p>
<ul>
-<li><p>Assign the encryption key to a bucket.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh bucket create -k encKey /vol/encryptedBucket</code></pre></div></li>
+<li>Assign the encryption key to a bucket.</li>
</ul>
-
-<p>After this command, all data written to the <em>encryptedBucket</em> will be encrypted
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh bucket create -k encKey /vol/encryptedBucket
+</span></span></code></pre></div><p>After this command, all data written to the <em>encryptedBucket</em> will be encrypted
via the encKey and while reading the clients will talk to Key Management
Server and read the key and decrypt it. In other words, the data stored
inside Ozone is always encrypted. The fact that data is encrypted at rest
diff --git a/docs/0.5.0-beta/security/securityacls.html b/docs/0.5.0-beta/security/securityacls.html
index e7bd7fb..c89aa71 100644
--- a/docs/0.5.0-beta/security/securityacls.html
+++ b/docs/0.5.0-beta/security/securityacls.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -229,26 +251,19 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone supports a set of native ACLs. These ACLs can be used independently or
along with Ranger. If Apache Ranger is enabled, then ACL will be checked
first with Ranger and then Ozone’s internal ACLs will be evaluated.</p>
-
<p>Ozone ACLs are a super set of Posix and S3 ACLs.</p>
-
<p>The general format of an ACL is <em>object</em>:<em>who</em>:<em>rights</em>.</p>
-
<p>Where an <em>object</em> can be:</p>
-
<ol>
<li><strong>Volume</strong> - An Ozone volume. e.g. <em>/volume</em></li>
<li><strong>Bucket</strong> - An Ozone bucket. e.g. <em>/volume/bucket</em></li>
<li><strong>Key</strong> - An object key or an object. e.g. <em>/volume/bucket/key</em></li>
<li><strong>Prefix</strong> - A path prefix for a specific key. e.g. <em>/volume/bucket/prefix1/prefix2</em></li>
</ol>
-
<p>Where a <em>who</em> can be:</p>
-
<ol>
<li><strong>User</strong> - A user in the Kerberos domain. User like in Posix world can be
named or unnamed.</li>
@@ -261,14 +276,11 @@ others in the Posix domain.</li>
the Posix semantics, This is needed for S3 protocol, where we express that
we have no way of knowing who the user is or we don’t care.</li>
</ol>
-
<div class="alert alert-success" role="alert">
A S3 user accesing Ozone via AWS v4 signature protocol will be translated
to the appropriate Kerberos user by Ozone Manager.
</div>
-
<p>Where a <em>right</em> can be:</p>
-
<ol>
<li><strong>Create</strong> – This ACL provides a user the ability to create buckets in a
volume and keys in a bucket. Please note: Under Ozone, Only admins can create volumes.</li>
@@ -282,12 +294,9 @@ allows the user to overwrite an existing ozone key.</li>
<li><strong>Read_ACL</strong> – Allows a user to read the ACL on a specific object.</li>
<li><strong>Write_ACL</strong> – Allows a user to write the ACL on a specific object.</li>
</ol>
-
<h3>Ozone Native ACL APIs</h3>
-
<p>The ACLs can be manipulated by a set of APIs supported by Ozone. The APIs
supported are:</p>
-
<ol>
<li><strong>SetAcl</strong> – This API will take user principal, the name, type
of the ozone object and a list of ACLs.</li>
diff --git a/docs/0.5.0-beta/shell.html b/docs/0.5.0-beta/shell.html
index d0908b6..6a956c0 100644
--- a/docs/0.5.0-beta/shell.html
+++ b/docs/0.5.0-beta/shell.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -225,7 +247,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">OzoneShell </h3>
diff --git a/docs/0.5.0-beta/shell/bucketcommands.html b/docs/0.5.0-beta/shell/bucketcommands.html
index d66ccb9..1b75b5e 100644
--- a/docs/0.5.0-beta/shell/bucketcommands.html
+++ b/docs/0.5.0-beta/shell/bucketcommands.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Bucket Commands</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,22 +255,16 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone shell supports the following bucket commands.</p>
-
<ul>
<li><a href="#create">create</a></li>
<li><a href="#delete">delete</a></li>
<li><a href="#info">info</a></li>
<li><a href="#list">list</a></li>
</ul>
-
<h3 id="create">Create</h3>
-
<p>The <code>bucket create</code> command allows users to create a bucket.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -258,37 +272,28 @@
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>-g, --enforcegdpr</td>
<td>Optional, if set to true it creates a GDPR compliant bucket, if not specified or set to false, it creates an ordinary bucket.</td>
</tr>
-
<tr>
<td>-k, --bucketKey</td>
<td>Optional, if a bucket encryption key name from the configured KMS server is specified, the files in the bucket will be transparently encrypted. Instruction on KMS configuration can be found from Hadoop KMS document.</td>
</tr>
-
<tr>
<td>Uri</td>
<td>The name of the bucket in <strong>/volume/bucket</strong> format.</td>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh bucket create /hive/jan</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh bucket create /hive/jan</span></span></code></pre></div>
<p>The above command will create a bucket called <em>jan</em> in the <em>hive</em> volume.
Since no scheme was specified this command defaults to O3 (RPC) protocol.</p>
-
<h3 id="delete">Delete</h3>
-
<p>The <code>bucket delete</code> command allows users to delete a bucket. If the
bucket is not empty then this command will fail.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -296,7 +301,6 @@ bucket is not empty then this command will fail.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
@@ -304,17 +308,11 @@ bucket is not empty then this command will fail.</p>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh bucket delete /hive/jan</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh bucket delete /hive/jan</span></span></code></pre></div>
<p>The above command will delete <em>jan</em> bucket if it is empty.</p>
-
<h3 id="info">Info</h3>
-
<p>The <code>bucket info</code> commands returns the information about the bucket.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -322,7 +320,6 @@ bucket is not empty then this command will fail.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
@@ -330,17 +327,11 @@ bucket is not empty then this command will fail.</p>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh bucket info /hive/jan</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh bucket info /hive/jan</span></span></code></pre></div>
<p>The above command will print out the information about <em>jan</em> bucket.</p>
-
<h3 id="list">List</h3>
-
<p>The <code>bucket list</code> command allows users to list the buckets in a volume.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -348,32 +339,26 @@ bucket is not empty then this command will fail.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>-l, --length</td>
<td>Maximum number of results to return. Default: 100</td>
</tr>
-
<tr>
<td>-p, --prefix</td>
<td>Optional, Only buckets that match this prefix will be returned.</td>
</tr>
-
<tr>
<td>-s, --start</td>
<td>The listing will start from key after the start key.</td>
</tr>
-
<tr>
<td>Uri</td>
<td>The name of the <em>volume</em>.</td>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh bucket list /hive</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh bucket list /hive</span></span></code></pre></div>
<p>This command will list all buckets on the volume <em>hive</em>.</p>
diff --git a/docs/0.5.0-beta/shell/format.html b/docs/0.5.0-beta/shell/format.html
index 9174826..e133131 100644
--- a/docs/0.5.0-beta/shell/format.html
+++ b/docs/0.5.0-beta/shell/format.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Shell Overview</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,48 +255,36 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone shell help can be invoked at <em>object</em> level or at <em>action</em> level.
For example:</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume --help</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume --help</span></span></code></pre></div>
<p>This will show all possible actions for volumes.</p>
-
<p>or it can be invoked to explain a specific action like
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume create --help</code></pre></div>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume create --help</span></span></code></pre></div>
This command will give you command line options of the create command.</p>
-
-<p></p></p>
-
+</p>
<h3 id="general-command-format">General Command Format</h3>
-
<p>The Ozone shell commands take the following format.</p>
-
<blockquote>
<p><em>ozone sh object action url</em></p>
</blockquote>
-
<p><strong>ozone</strong> script is used to invoke all Ozone sub-commands. The ozone shell is
invoked via <code>sh</code> command.</p>
-
<p>The object can be a volume, bucket or a key. The action is various verbs like
create, list, delete etc.</p>
-
<p>Ozone URL can point to a volume, bucket or keys in the following format:</p>
-
<p><em>[schema][server:port]/volume/bucket/key</em></p>
-
<p>Where,</p>
-
<ol>
-<li><p><strong>Schema</strong> - This should be <code>o3</code> which is the native RPC protocol to access
-Ozone API. The usage of the schema is optional.</p></li>
-
-<li><p><strong>Server:Port</strong> - This is the address of the Ozone Manager. If the port is
-omitted the default port from ozone-site.xml will be used.</p></li>
+<li>
+<p><strong>Schema</strong> - This should be <code>o3</code> which is the native RPC protocol to access
+Ozone API. The usage of the schema is optional.</p>
+</li>
+<li>
+<p><strong>Server:Port</strong> - This is the address of the Ozone Manager. If the port is
+omitted the default port from ozone-site.xml will be used.</p>
+</li>
</ol>
-
<p>Depending on the call, the volume/bucket/key names will be part of the URL.
Please see volume commands, bucket commands, and key commands section for more
detail.</p>
diff --git a/docs/0.5.0-beta/shell/index.xml b/docs/0.5.0-beta/shell/index.xml
index 388a62a..d9d3a0c 100644
--- a/docs/0.5.0-beta/shell/index.xml
+++ b/docs/0.5.0-beta/shell/index.xml
@@ -1,15 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Command Line Interface on Ozone</title>
<link>/shell.html</link>
<description>Recent content in Command Line Interface on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
-
- <atom:link href="/shell/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/shell/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Shell Overview</title>
<link>/shell/format.html</link>
@@ -47,4 +43,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/shell/keycommands.html b/docs/0.5.0-beta/shell/keycommands.html
index 0b632f4..7bf40a7 100644
--- a/docs/0.5.0-beta/shell/keycommands.html
+++ b/docs/0.5.0-beta/shell/keycommands.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Key Commands</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,9 +255,7 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone shell supports the following key commands.</p>
-
<ul>
<li><a href="#get">get</a></li>
<li><a href="#put">put</a></li>
@@ -246,13 +264,9 @@
<li><a href="#list">list</a></li>
<li><a href="#rename">rename</a></li>
</ul>
-
<h3 id="get">Get</h3>
-
<p>The <code>key get</code> command downloads a key from Ozone cluster to local file system.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -260,30 +274,23 @@
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
<td>The name of the key in <strong>/volume/bucket/key</strong> format.</td>
</tr>
-
<tr>
<td>FileName</td>
<td>Local file to download the key to.</td>
</tr>
</tbody>
</table>
-
-<p><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh key get /hive/jan/sales.orc sales.orc</code></pre></div>
+<p><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh key get /hive/jan/sales.orc sales.orc</span></span></code></pre></div>
Downloads the file sales.orc from the <em>/hive/jan</em> bucket and writes to the
local file sales.orc.</p>
-
<h3 id="put">Put</h3>
-
<p>The <code>key put</code> command uploads a file from the local file system to the specified bucket.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -291,34 +298,26 @@ local file sales.orc.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
<td>The name of the key in <strong>/volume/bucket/key</strong> format.</td>
</tr>
-
<tr>
<td>FileName</td>
<td>Local file to upload.</td>
</tr>
-
<tr>
<td>-r, --replication</td>
<td>Optional, Number of copies, ONE or THREE are the options. Picks up the default from cluster configuration.</td>
</tr>
</tbody>
</table>
-
-<p><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh key put /hive/jan/corrected-sales.orc sales.orc</code></pre></div>
+<p><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh key put /hive/jan/corrected-sales.orc sales.orc</span></span></code></pre></div>
The above command will put the sales.orc as a new key into <em>/hive/jan/corrected-sales.orc</em>.</p>
-
<h3 id="delete">Delete</h3>
-
<p>The <code>key delete</code> command removes the key from the bucket.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -326,7 +325,6 @@ The above command will put the sales.orc as a new key into <em>/hive/jan/correct
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
@@ -334,17 +332,11 @@ The above command will put the sales.orc as a new key into <em>/hive/jan/correct
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh key delete /hive/jan/corrected-sales.orc</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh key delete /hive/jan/corrected-sales.orc</span></span></code></pre></div>
<p>The above command deletes the key <em>/hive/jan/corrected-sales.orc</em>.</p>
-
<h3 id="info">Info</h3>
-
<p>The <code>key info</code> commands returns the information about the key.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -352,7 +344,6 @@ The above command will put the sales.orc as a new key into <em>/hive/jan/correct
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
@@ -360,18 +351,12 @@ The above command will put the sales.orc as a new key into <em>/hive/jan/correct
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh key info /hive/jan/sales.orc</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh key info /hive/jan/sales.orc</span></span></code></pre></div>
<p>The above command will print out the information about <em>/hive/jan/sales.orc</em>
key.</p>
-
<h3 id="list">List</h3>
-
<p>The <code>key list</code> command allows user to list all keys in a bucket.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -379,40 +364,30 @@ key.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>-l, --length</td>
<td>Maximum number of results to return. Default: 100</td>
</tr>
-
<tr>
<td>-p, --prefix</td>
<td>Optional, Only keys that match this prefix will be returned.</td>
</tr>
-
<tr>
<td>-s, --start</td>
<td>The listing will start from key after the start key.</td>
</tr>
-
<tr>
<td>Uri</td>
<td>The name of the <em>volume</em>.</td>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh key list /hive/jan</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh key list /hive/jan</span></span></code></pre></div>
<p>This command will list all keys in the bucket <em>/hive/jan</em>.</p>
-
<h3 id="rename">Rename</h3>
-
<p>The <code>key rename</code> command changes the name of an existing key in the specified bucket.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -420,26 +395,22 @@ key.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
<td>The name of the bucket in <strong>/volume/bucket</strong> format.</td>
</tr>
-
<tr>
<td>FromKey</td>
<td>The existing key to be renamed</td>
</tr>
-
<tr>
<td>ToKey</td>
<td>The new desired name of the key</td>
</tr>
</tbody>
</table>
-
-<p><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh key rename /hive/jan sales.orc new_name.orc</code></pre></div>
+<p><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh key rename /hive/jan sales.orc new_name.orc</span></span></code></pre></div>
The above command will rename <em>sales.orc</em> to <em>new_name.orc</em> in the bucket <em>/hive/jan</em>.</p>
diff --git a/docs/0.5.0-beta/shell/volumecommands.html b/docs/0.5.0-beta/shell/volumecommands.html
index 665988b..7058c34 100644
--- a/docs/0.5.0-beta/shell/volumecommands.html
+++ b/docs/0.5.0-beta/shell/volumecommands.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Volume Commands</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,9 +255,7 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Volume commands generally need administrator privileges. The ozone shell supports the following volume commands.</p>
-
<ul>
<li><a href="#create">create</a></li>
<li><a href="#delete">delete</a></li>
@@ -245,14 +263,10 @@
<li><a href="#list">list</a></li>
<li><a href="#update">update</a></li>
</ul>
-
<h3 id="create">Create</h3>
-
<p>The <code>volume create</code> command allows an administrator to create a volume and
assign it to a user.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -260,37 +274,28 @@ assign it to a user.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>-q, --quota</td>
<td>Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster.</td>
</tr>
-
<tr>
<td>-u, --user</td>
<td>Required, The name of the user who owns this volume. This user can create, buckets and keys on this volume.</td>
</tr>
-
<tr>
<td>Uri</td>
<td>The name of the volume.</td>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume create --quota<span style="color:#f92672">=</span>1TB --user<span style="color:#f92672">=</span>bilbo /hive</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume create --quota<span style="color:#f92672">=</span>1TB --user<span style="color:#f92672">=</span>bilbo /hive</span></span></code></pre></div>
<p>The above command will create a volume called <em>hive</em> on the ozone cluster. This
volume has a quota of 1TB, and the owner is <em>bilbo</em>.</p>
-
<h3 id="delete">Delete</h3>
-
<p>The <code>volume delete</code> command allows an administrator to delete a volume. If the
volume is not empty then this command will fail.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -298,7 +303,6 @@ volume is not empty then this command will fail.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
@@ -306,19 +310,13 @@ volume is not empty then this command will fail.</p>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume delete /hive</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume delete /hive</span></span></code></pre></div>
<p>The above command will delete the volume hive, if the volume has no buckets
inside it.</p>
-
<h3 id="info">Info</h3>
-
<p>The <code>volume info</code> commands returns the information about the volume including
quota and owner information.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -326,7 +324,6 @@ quota and owner information.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>Uri</td>
@@ -334,25 +331,15 @@ quota and owner information.</p>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume info /hive</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume info /hive</span></span></code></pre></div>
<p>The above command will print out the information about hive volume.</p>
-
<h3 id="list">List</h3>
-
<p>The <code>volume list</code> command will list the volumes owned by a user.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume list --user hadoop</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume list --user hadoop</span></span></code></pre></div>
<p>The above command will print out all the volumes owned by the user hadoop.</p>
-
<h3 id="update">Update</h3>
-
<p>The volume update command allows changing of owner and quota on a given volume.</p>
-
-<p><strong><em>Params:</em></strong></p>
-
+<p><em><strong>Params:</strong></em></p>
<table>
<thead>
<tr>
@@ -360,27 +347,22 @@ quota and owner information.</p>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>-q, --quota</td>
<td>Optional, This argument that specifies the maximum size this volume can use in the Ozone cluster.</td>
</tr>
-
<tr>
<td>-u, --user</td>
<td>Optional, The name of the user who owns this volume. This user can create, buckets and keys on this volume.</td>
</tr>
-
<tr>
<td>Uri</td>
<td>The name of the volume.</td>
</tr>
</tbody>
</table>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume update --quota<span style="color:#f92672">=</span>10TB /hive</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume update --quota<span style="color:#f92672">=</span>10TB /hive</span></span></code></pre></div>
<p>The above command updates the volume quota to 10TB.</p>
diff --git a/docs/0.5.0-beta/sitemap.xml b/docs/0.5.0-beta/sitemap.xml
index 2987e0b..789b9b5 100644
--- a/docs/0.5.0-beta/sitemap.xml
+++ b/docs/0.5.0-beta/sitemap.xml
@@ -1,18 +1,18 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
-
- <sitemap>
- <loc>en/sitemap.xml</loc>
-
- <lastmod>2019-06-07T00:00:00+00:00</lastmod>
-
- </sitemap>
-
- <sitemap>
- <loc>zh/sitemap.xml</loc>
-
- <lastmod>2017-10-10T00:00:00+00:00</lastmod>
-
- </sitemap>
-
+
+ <sitemap>
+ <loc>en/sitemap.xml</loc>
+
+ <lastmod>2020-01-25T00:54:46+08:00</lastmod>
+
+ </sitemap>
+
+ <sitemap>
+ <loc>zh/sitemap.xml</loc>
+
+ <lastmod>2020-02-22T06:23:05+08:00</lastmod>
+
+ </sitemap>
+
</sitemapindex>
diff --git a/docs/0.5.0-beta/start.html b/docs/0.5.0-beta/start.html
index b632ed0..f3777f5 100644
--- a/docs/0.5.0-beta/start.html
+++ b/docs/0.5.0-beta/start.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -225,7 +247,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">Installing Ozone </h3>
@@ -239,16 +260,12 @@ Kubernetes or bare-metal.
</p>
</div>
</div>
-
<section class="row cardgroup">
-
-<span class="label label-warning label-">Easy Start</span>
-
+<p><span class="label label-warning label-">Easy Start</span></p>
<h2>Running Ozone from Docker Hub</h2>
-
-You can try out Ozone using docker hub without downloading the official release. This makes it easy to explore Ozone.
+<p>You can try out Ozone using docker hub without downloading the official release. This makes it easy to explore Ozone.
<br />
-
+
<div class="col-sm-6">
@@ -274,20 +291,14 @@ You can try out Ozone using docker hub without downloading the official release.
</div>
</div>
</div>
-
-
+</p>
</section>
-
<section class="row cardgroup">
-
-<span class="label label-success">Recommended</span>
-
-
+<p><span class="label label-success">Recommended</span></p>
<h2>Running Ozone from an Official Release</h2>
-
- Apache Ozone can also be run from the official release packages. Along with the official source releases, we also release a set of convenience binary packages. It is easy to run these binaries in different configurations.
+<p>Apache Ozone can also be run from the official release packages. Along with the official source releases, we also release a set of convenience binary packages. It is easy to run these binaries in different configurations.
<br />
-
+
<div class="col-sm-6">
@@ -312,10 +323,9 @@ Ozone is designed to work concurrently with HDFS. The physical cluster instructi
</div>
</div>
</div>
+</p>
-
-
<div class="col-sm-6">
@@ -341,7 +351,6 @@ Ozone is designed to work well under Kubernetes. These are instructions to deplo
</div>
-
<div class="col-sm-6">
@@ -368,7 +377,6 @@ Ozone comes with a standard set of K8s resources. You can deploy them to MiniKub
</div>
-
<div class="col-sm-6">
@@ -394,18 +402,12 @@ Ozone comes with a standard set of K8s resources. You can deploy them to MiniKub
</div>
</div>
-
</section>
-
<section class="row cardgroup">
-
-<span class="label label-danger">Hadoop Ninja</span>
-
+<p><span class="label label-danger">Hadoop Ninja</span></p>
<h2>Building From Sources </h2>
+<p>Instructions to build Ozone from source to create deployment packages.</p>
- Instructions to build Ozone from source to create deployment packages.
-
-
<div class="col-sm-6">
@@ -431,7 +433,6 @@ If you are a Hadoop ninja, and wise in the ways of Apache, you already know that
</div>
</div>
-
</section>
diff --git a/docs/0.5.0-beta/start/fromsource.html b/docs/0.5.0-beta/start/fromsource.html
index ba1ff70..b27cafb 100644
--- a/docs/0.5.0-beta/start/fromsource.html
+++ b/docs/0.5.0-beta/start/fromsource.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>From Source</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -236,7 +256,6 @@
limitations under the License.
-->
-
<div class="panel panel-default">
<div class="panel-heading">Requirements</div>
<div class="panel-body">
@@ -245,39 +264,34 @@
<li>Maven</li>
<li>Protoc (2.5)</li>
</ul>
+
</div>
</div>
-
<div class="alert alert-info" role="alert">This is a guide on how to build the ozone sources. If you are <font
color="red">not</font>
planning to build sources yourself, you can safely skip this page.</div>
-
<p>If you are a Hadoop ninja, and wise in the ways of Apache, you already know
that a real Apache release is a source release.</p>
-
<p>If you want to build from sources, Please untar the source tarball and run
the ozone build command. This instruction assumes that you have all the
dependencies to build Hadoop on your build machine. If you need instructions
on how to build Hadoop, please look at the Apache Hadoop Website.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">mvn clean package -DskipTests<span style="color:#f92672">=</span>true</code></pre></div>
-<p>This will build an ozone-<version>.tar.gz in your <code>hadoop-ozone/dist/target</code> directory.</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>mvn clean package -DskipTests<span style="color:#f92672">=</span>true
+</span></span></code></pre></div><p>This will build an ozone-<version>.tar.gz in your <code>hadoop-ozone/dist/target</code> directory.</p>
<p>You can copy this tarball and use this instead of binary artifacts that are
provided along with the official release.</p>
-
<h2 id="how-to-test-the-build">How to test the build</h2>
-
<p>You can run the acceptance tests in the hadoop-ozone directory to make sure
that your build is functional. To launch the acceptance tests, please follow
- the instructions in the <strong>README.md</strong> in the <code>smoketest</code> directory.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd smoketest
-./test.sh</code></pre></div>
-<p>You can also execute only a minimal subset of the tests:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd smoketest
-./test.sh --env ozone basic</code></pre></div>
-<p>Acceptance tests will start a small ozone cluster and verify that ozone shell and ozone file
- system is fully functional.</p>
+the instructions in the <strong>README.md</strong> in the <code>smoketest</code> directory.</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd smoketest
+</span></span><span style="display:flex;"><span>./test.sh
+</span></span></code></pre></div><p>You can also execute only a minimal subset of the tests:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd smoketest
+</span></span><span style="display:flex;"><span>./test.sh --env ozone basic
+</span></span></code></pre></div><p>Acceptance tests will start a small ozone cluster and verify that ozone shell and ozone file
+system is fully functional.</p>
diff --git a/docs/0.5.0-beta/start/index.xml b/docs/0.5.0-beta/start/index.xml
index 172e1dd..5182ae0 100644
--- a/docs/0.5.0-beta/start/index.xml
+++ b/docs/0.5.0-beta/start/index.xml
@@ -1,22 +1,18 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Getting Started on Ozone</title>
<link>/start.html</link>
<description>Recent content in Getting Started on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
-
- <atom:link href="/start/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/start/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Simple Single Ozone</title>
<link>/start/startfromdockerhub.html</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/startfromdockerhub.html</guid>
- <description>Requirements Working docker setup AWS CLI (optional) Ozone in a Single Container The easiest way to start up an all-in-one ozone container is to use the latest docker image from docker hub:
+ <description>Requirements Working docker setup AWS CLI (optional) Ozone in a Single Container The easiest way to start up an all-in-one ozone container is to use the latest docker image from docker hub:
docker run -p 9878:9878 -p 9876:9876 apache/ozone This command will pull down the ozone image from docker hub and start all ozone services in a single container. This container will run the required metadata servers (Ozone Manager, Storage Container Manager) one data node and the S3 compatible REST server (S3 Gateway).</description>
</item>
@@ -27,7 +23,7 @@ docker run -p 9878:9878 -p 9876:9876 apache/ozone This command will pull down th
<guid>/start/onprem.html</guid>
<description>If you are feeling adventurous, you can setup ozone in a real cluster. Setting up a real cluster requires us to understand the components of Ozone. Ozone is designed to work concurrently with HDFS. However, Ozone is also capable of running independently. The components of ozone are the same in both approaches.
-Ozone Components Ozone Manager - Is the server that is in charge of the namespace of Ozone.</description>
+Ozone Components Ozone Manager - Is the server that is in charge of the namespace of Ozone. Ozone Manager is responsible for all volume, bucket and key operations.</description>
</item>
<item>
@@ -36,10 +32,10 @@ Ozone Components Ozone Manager - Is the server that is in charge of the namespa
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/minikube.html</guid>
- <description>Requirements Working minikube setup kubectl kubernetes/examples folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. By default the kubernetes resource files are configured to use apache/ozone image from the dockerhub.
+ <description>Requirements Working minikube setup kubectl kubernetes/examples folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. By default the kubernetes resource files are configured to use apache/ozone image from the dockerhub.
To deploy it to minikube use the minikube configuration set:
-cd kubernetes/examples/minikube kubectl apply -f . And you can check the results with
-kubectl get pod Note: the kubernetes/examples/minikube resource set is optimized for minikube usage:</description>
+cd kubernetes/examples/minikube kubectl apply -f . And you can check the results with
+kubectl get pod Note: the kubernetes/examples/minikube resource set is optimized for minikube usage:</description>
</item>
<item>
@@ -48,7 +44,7 @@ kubectl get pod Note: the kubernetes/examples/minikube resource set is optimize
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/kubernetes.html</guid>
- <description>Requirements Working kubernetes cluster (LoadBalancer, PersistentVolume are not required) kubectl As the apache/ozone docker images are available from the dockerhub the deployment process is very similar to Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host) Deploy to kubernetes
+ <description>Requirements Working kubernetes cluster (LoadBalancer, PersistentVolume are not required) kubectl As the apache/ozone docker images are available from the dockerhub the deployment process is very similar to Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host) Deploy to kubernetes
kubernetes/examples folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases.</description>
</item>
@@ -58,9 +54,9 @@ kubernetes/examples folder of the ozone distribution contains kubernetes deploym
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/runningviadocker.html</guid>
- <description>Requirements docker and docker-compose Download the Ozone binary tarball and untar it.
- Go to the directory where the docker compose files exist and tell docker-compose to start Ozone in the background. This will start a small ozone instance on your machine.
- cd compose/ozone/ docker-compose up -d To verify that ozone is working as expected, let us log into a data node and run freon, the load generator for Ozone.</description>
+ <description>Requirements docker and docker-compose Download the Ozone binary tarball and untar it.
+Go to the directory where the docker compose files exist and tell docker-compose to start Ozone in the background. This will start a small ozone instance on your machine.
+cd compose/ozone/ docker-compose up -d To verify that ozone is working as expected, let us log into a data node and run freon, the load generator for Ozone. The exec datanode bash command will open a bash shell on the datanode.</description>
</item>
<item>
@@ -69,9 +65,9 @@ kubernetes/examples folder of the ozone distribution contains kubernetes deploym
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/start/fromsource.html</guid>
- <description>Requirements Java 1.8 Maven Protoc (2.5) This is a guide on how to build the ozone sources. If you are not planning to build sources yourself, you can safely skip this page. If you are a Hadoop ninja, and wise in the ways of Apache, you already know that a real Apache release is a source release.
+ <description>Requirements Java 1.8 Maven Protoc (2.5) This is a guide on how to build the ozone sources. If you are not planning to build sources yourself, you can safely skip this page. If you are a Hadoop ninja, and wise in the ways of Apache, you already know that a real Apache release is a source release.
If you want to build from sources, Please untar the source tarball and run the ozone build command.</description>
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/start/kubernetes.html b/docs/0.5.0-beta/start/kubernetes.html
index 6a6c980..b91bbbc 100644
--- a/docs/0.5.0-beta/start/kubernetes.html
+++ b/docs/0.5.0-beta/start/kubernetes.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -234,7 +256,6 @@
limitations under the License.
-->
-
<div class="panel panel-default">
<div class="panel-heading">Requirements</div>
<div class="panel-body">
@@ -242,34 +263,24 @@
<li>Working kubernetes cluster (LoadBalancer, PersistentVolume are not required)</li>
<li>kubectl</li>
</ul>
+
</div>
</div>
-
<p>As the <em>apache/ozone</em> docker images are available from the dockerhub the deployment process is very similar to Minikube deployment. The only big difference is that we have dedicated set of k8s files for hosted clusters (for example we can use one datanode per host)
Deploy to kubernetes</p>
-
<p><code>kubernetes/examples</code> folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases.</p>
-
<p>To deploy to a hosted cluster use the ozone subdirectory:</p>
-
-<pre><code>cd kubernetes/examples/ozone
+<pre tabindex="0"><code>cd kubernetes/examples/ozone
kubectl apply -f .
-</code></pre>
-
-<p>And you can check the results with</p>
-
-<pre><code>kubectl get pod
+</code></pre><p>And you can check the results with</p>
+<pre tabindex="0"><code>kubectl get pod
Access the services
-</code></pre>
-
-<p>Now you can access any of the services. By default the services are not published but you can access them with port-foward rules.</p>
-
-<pre><code>kubectl port-forward s3g-0 9878:9878
+</code></pre><p>Now you can access any of the services. By default the services are not published but you can access them with port-foward rules.</p>
+<pre tabindex="0"><code>kubectl port-forward s3g-0 9878:9878
kubectl port-forward scm-0 9876:9876
</code></pre>
-
<a class="btn btn-success btn-lg" href="../start/runningviadocker.html">Next >></a>
diff --git a/docs/0.5.0-beta/start/minikube.html b/docs/0.5.0-beta/start/minikube.html
index c14cf01..eb91aba 100644
--- a/docs/0.5.0-beta/start/minikube.html
+++ b/docs/0.5.0-beta/start/minikube.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Minikube & Ozone</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -236,7 +256,6 @@
limitations under the License.
-->
-
<div class="panel panel-default">
<div class="panel-heading">Requirements</div>
<div class="panel-body">
@@ -244,50 +263,38 @@
<li>Working minikube setup</li>
<li>kubectl</li>
</ul>
+
</div>
</div>
-
<p><code>kubernetes/examples</code> folder of the ozone distribution contains kubernetes deployment resource files for multiple use cases. By default the kubernetes resource files are configured to use <code>apache/ozone</code> image from the dockerhub.</p>
-
<p>To deploy it to minikube use the minikube configuration set:</p>
-
-<pre><code>cd kubernetes/examples/minikube
+<pre tabindex="0"><code>cd kubernetes/examples/minikube
kubectl apply -f .
-</code></pre>
-
-<p>And you can check the results with</p>
-
-<pre><code>kubectl get pod
-</code></pre>
-
-<p>Note: the kubernetes/examples/minikube resource set is optimized for minikube usage:</p>
-
+</code></pre><p>And you can check the results with</p>
+<pre tabindex="0"><code>kubectl get pod
+</code></pre><p>Note: the kubernetes/examples/minikube resource set is optimized for minikube usage:</p>
<ul>
<li>You can have multiple datanodes even if you have only one host (in a real production cluster usually you need one datanode per physical host)</li>
<li>The services are published with node port</li>
</ul>
-
<h2 id="access-the-services">Access the services</h2>
-
<p>Now you can access any of the services. For each web endpoint an additional NodeType service is defined in the minikube k8s resource set. NodeType services are available via a generated port of any of the host nodes:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">kubectl get svc
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT<span style="color:#f92672">(</span>S<span style="color:#f92672">)</span> AGE
-datanode ClusterIP None <none> <none> 27s
-kubernetes ClusterIP <span style="color:#ae81ff">10</span>.96.0.1 <none> <span style="color:#ae81ff">443</span>/TCP 118m
-om ClusterIP None <none> <span style="color:#ae81ff">9874</span>/TCP 27s
-om-public NodePort <span style="color:#ae81ff">10</span>.108.48.148 <none> <span style="color:#ae81ff">9874</span>:32649/TCP 27s
-s3g ClusterIP None <none> <span style="color:#ae81ff">9878</span>/TCP 27s
-s3g-public NodePort <span style="color:#ae81ff">10</span>.97.133.137 <none> <span style="color:#ae81ff">9878</span>:31880/TCP 27s
-scm ClusterIP None <none> <span style="color:#ae81ff">9876</span>/TCP 27s
-scm-public NodePort <span style="color:#ae81ff">10</span>.105.231.28 <none> <span style="color:#ae81ff">9876</span>:32171/TCP 27s</code></pre></div>
-<p>Minikube contains a convenience command to access any of the NodePort services:</p>
-
-<pre><code>minikube service s3g-public
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl get svc
+</span></span><span style="display:flex;"><span>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT<span style="color:#f92672">(</span>S<span style="color:#f92672">)</span> AGE
+</span></span><span style="display:flex;"><span>datanode ClusterIP None <none> <none> 27s
+</span></span><span style="display:flex;"><span>kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 118m
+</span></span><span style="display:flex;"><span>om ClusterIP None <none> 9874/TCP 27s
+</span></span><span style="display:flex;"><span>om-public NodePort 10.108.48.148 <none> 9874:32649/TCP 27s
+</span></span><span style="display:flex;"><span>s3g ClusterIP None <none> 9878/TCP 27s
+</span></span><span style="display:flex;"><span>s3g-public NodePort 10.97.133.137 <none> 9878:31880/TCP 27s
+</span></span><span style="display:flex;"><span>scm ClusterIP None <none> 9876/TCP 27s
+</span></span><span style="display:flex;"><span>scm-public NodePort 10.105.231.28 <none> 9876:32171/TCP 27s
+</span></span></code></pre></div><p>Minikube contains a convenience command to access any of the NodePort services:</p>
+<pre tabindex="0"><code>minikube service s3g-public
Opening kubernetes service default/s3g-public in default browser...
</code></pre>
-
<a class="btn btn-success btn-lg" href="../start/kubernetes.html">Next >></a>
diff --git a/docs/0.5.0-beta/start/onprem.html b/docs/0.5.0-beta/start/onprem.html
index 858c84b..120555d 100644
--- a/docs/0.5.0-beta/start/onprem.html
+++ b/docs/0.5.0-beta/start/onprem.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Ozone On Premise Installation</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -235,55 +255,49 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>If you are feeling adventurous, you can setup ozone in a real cluster.
Setting up a real cluster requires us to understand the components of Ozone.
Ozone is designed to work concurrently with HDFS. However, Ozone is also
capable of running independently. The components of ozone are the same in both approaches.</p>
-
<h2 id="ozone-components">Ozone Components</h2>
-
<ol>
<li>Ozone Manager - Is the server that is in charge of the namespace of Ozone. Ozone Manager is responsible for all volume, bucket and key operations.</li>
<li>Storage Container Manager - Acts as the block manager. Ozone Manager
requests blocks from SCM, to which clients can write data.</li>
<li>Datanodes - Ozone data node code runs inside the HDFS datanode or in the independent deployment case runs an ozone datanode daemon.</li>
</ol>
-
<h2 id="setting-up-an-ozone-only-cluster">Setting up an Ozone only cluster</h2>
-
<ul>
-<li><p>Please untar the ozone-<version> to the directory where you are going
+<li>
+<p>Please untar the ozone-<version> to the directory where you are going
to run Ozone from. We need Ozone jars on all machines in the cluster. So you
-need to do this on all machines in the cluster.</p></li>
-
-<li><p>Ozone relies on a configuration file called <code>ozone-site.xml</code>. To
+need to do this on all machines in the cluster.</p>
+</li>
+<li>
+<p>Ozone relies on a configuration file called <code>ozone-site.xml</code>. To
generate a template that you can replace with proper values, please run the
following command. This will generate a template called <code>ozone-site.xml</code> at
-the specified path (directory).</p></li>
+the specified path (directory).</p>
+</li>
</ul>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone genconf <path></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone genconf <path></span></span></code></pre></div>
<p>Let us look at the settings inside the generated file (ozone-site.xml) and
how they control ozone. Once the right values are defined, this file
needs to be copied to <code>ozone directory/etc/hadoop</code>.</p>
-
<ul>
-<li><strong>ozone.metadata.dirs</strong> Allows Administrators to specify where the
+<li>
+<p><strong>ozone.metadata.dirs</strong> Allows Administrators to specify where the
metadata must reside. Usually you pick your fastest disk (SSD if
you have them on your nodes). OzoneManager, SCM and datanode will write the
metadata to this path. This is a required setting, if this is missing Ozone
-will fail to come up.</li>
-</ul>
-
+will fail to come up.</p>
<p>Here is an example,</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.metadata.dirs<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>/data/disk1/meta<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span></code></pre></div>
-
+</li>
+</ul>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.metadata.dirs<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>/data/disk1/meta<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span></span></span></code></pre></div>
<ul>
<li><strong>ozone.scm.names</strong> Storage container manager(SCM) is a distributed block
service which is used by ozone. This property allows data nodes to discover
@@ -291,40 +305,31 @@ SCM’s address. Data nodes send heartbeat to SCM.
Until HA feature is complete, we configure ozone.scm.names to be a
single machine.</li>
</ul>
-
<p>Here is an example,</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.scm.names<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>scm.hadoop.apache.org<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span>
- </code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.scm.names<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>scm.hadoop.apache.org<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span></span></span></code></pre></div>
<ul>
<li><strong>ozone.scm.datanode.id.dir</strong> Data nodes generate a Unique ID called Datanode
ID. This identity is written to the file datanode.id in a directory specified by this path. <em>Data nodes
will create this path if it doesn’t exist already.</em></li>
</ul>
-
<p>Here is an example,
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.scm.datanode.id.dir<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>/data/disk1/meta/node<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span></code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.scm.datanode.id.dir<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>/data/disk1/meta/node<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span></span></span></code></pre></div></p>
<ul>
<li><strong>ozone.om.address</strong> OM server address. This is used by OzoneClient and
Ozone File System.</li>
</ul>
-
<p>Here is an example,
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.om.address<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>ozonemanager.hadoop.apache.org<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span></code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.om.address<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>ozonemanager.hadoop.apache.org<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span></span></span></code></pre></div></p>
<h2 id="ozone-settings-summary">Ozone Settings Summary</h2>
-
<table>
<thead>
<tr>
@@ -333,38 +338,32 @@ Ozone File System.</li>
<th>Comment</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>ozone.metadata.dirs</td>
<td>file path</td>
<td>The metadata will be stored here.</td>
</tr>
-
<tr>
<td>ozone.scm.names</td>
<td>SCM server name</td>
<td>Hostname:port or IP:port address of SCM.</td>
</tr>
-
<tr>
<td>ozone.scm.block.client.address</td>
<td>SCM server name and port</td>
<td>Used by services like OM</td>
</tr>
-
<tr>
<td>ozone.scm.client.address</td>
<td>SCM server name and port</td>
<td>Used by client-side</td>
</tr>
-
<tr>
<td>ozone.scm.datanode.address</td>
<td>SCM server name and port</td>
<td>Used by datanode to talk to SCM</td>
</tr>
-
<tr>
<td>ozone.om.address</td>
<td>OM server name</td>
@@ -372,42 +371,28 @@ Ozone File System.</li>
</tr>
</tbody>
</table>
-
<h2 id="startup-the-cluster">Startup the cluster</h2>
-
<p>Before we boot up the Ozone cluster, we need to initialize both SCM and Ozone Manager.</p>
-
-<p><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone scm --init</code></pre></div>
+<p><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone scm --init</span></span></code></pre></div>
This allows SCM to create the cluster Identity and initialize its state.
The <code>init</code> command is similar to Namenode format. Init command is executed only once, that allows SCM to create all the required on-disk structures to work correctly.
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone --daemon start scm</code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone --daemon start scm</span></span></code></pre></div></p>
<p>Once we know SCM is up and running, we can create an Object Store for our use. This is done by running the following command.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone om --init</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone om --init</span></span></code></pre></div>
<p>Once Ozone manager is initialized, we are ready to run the name service.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone --daemon start om</code></pre></div>
-
-<p>At this point Ozone’s name services, the Ozone manager, and the block service SCM is both running.<br />
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone --daemon start om</span></span></code></pre></div>
+<p>At this point Ozone’s name services, the Ozone manager, and the block service SCM is both running.<br>
<strong>Please note</strong>: If SCM is not running
<code>om --init</code> command will fail. SCM start will fail if on-disk data structures are missing. So please make sure you have done both <code>scm --init</code> and <code>om --init</code> commands.</p>
-
<p>Now we need to start the data nodes. Please run the following command on each datanode.
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone --daemon start datanode</code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone --daemon start datanode</span></span></code></pre></div></p>
<p>At this point SCM, Ozone Manager and data nodes are up and running.</p>
-
-<p><strong><em>Congratulations!, You have set up a functional ozone cluster.</em></strong></p>
-
+<p><em><strong>Congratulations!, You have set up a functional ozone cluster.</strong></em></p>
<h2 id="shortcut">Shortcut</h2>
-
<p>If you want to make your life simpler, you can just run
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone scm --init
-ozone om --init
-start-ozone.sh</code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone scm --init
+</span></span><span style="display:flex;"><span>ozone om --init
+</span></span><span style="display:flex;"><span>start-ozone.sh</span></span></code></pre></div></p>
<p>This assumes that you have set up the slaves file correctly and ssh
configuration that allows ssh-ing to all data nodes. This is the same as the
HDFS configuration, so please refer to HDFS documentation on how to set this
diff --git a/docs/0.5.0-beta/start/runningviadocker.html b/docs/0.5.0-beta/start/runningviadocker.html
index f365dc9..e68aa23 100644
--- a/docs/0.5.0-beta/start/runningviadocker.html
+++ b/docs/0.5.0-beta/start/runningviadocker.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -234,47 +256,42 @@
limitations under the License.
-->
-
<div class="panel panel-default">
<div class="panel-heading">Requirements</div>
<div class="panel-body">
<ul>
<li>docker and docker-compose</li>
</ul>
+
</div>
</div>
-
<ul>
-<li><p>Download the Ozone binary tarball and untar it.</p></li>
-
-<li><p>Go to the directory where the docker compose files exist and tell
+<li>
+<p>Download the Ozone binary tarball and untar it.</p>
+</li>
+<li>
+<p>Go to the directory where the docker compose files exist and tell
<code>docker-compose</code> to start Ozone in the background. This will start a small
-ozone instance on your machine.</p></li>
+ozone instance on your machine.</p>
+</li>
</ul>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">cd compose/ozone/
-
-docker-compose up -d</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>cd compose/ozone/
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span>docker-compose up -d</span></span></code></pre></div>
<p>To verify that ozone is working as expected, let us log into a data node and
run <em>freon</em>, the load generator for Ozone. The <code>exec datanode bash</code> command
will open a bash shell on the datanode.</p>
-
<p>The <code>ozone freon</code> command is executed within the datanode container. You can quit freon via CTRL-C any time. The
<code>rk</code> profile instructs freon to generate random keys.</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose exec datanode bash
-ozone freon rk</code></pre></div>
-
-<p>You can check out the <strong>OzoneManager UI</strong> at <a href="http://localhost:9874/">http://localhost:9874/</a> to see the
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose exec datanode bash
+</span></span><span style="display:flex;"><span>ozone freon rk</span></span></code></pre></div>
+<p>You can check out the <strong>OzoneManager UI</strong> at http://localhost:9874/ to see the
activity generated by freon.
While you are there, please don’t forget to check out the ozone configuration explorer.</p>
-
-<p><strong><em>Congratulations, You have just run your first ozone cluster.</em></strong></p>
-
+<p><em><strong>Congratulations, You have just run your first ozone cluster.</strong></em></p>
<p>To shutdown the cluster, please run
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose down</code></pre></div></p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose down</span></span></code></pre></div></p>
diff --git a/docs/0.5.0-beta/start/startfromdockerhub.html b/docs/0.5.0-beta/start/startfromdockerhub.html
index 38fc1af..9deaffe 100644
--- a/docs/0.5.0-beta/start/startfromdockerhub.html
+++ b/docs/0.5.0-beta/start/startfromdockerhub.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -217,9 +239,7 @@
<div class="col-md-9">
<h1>Simple Single Ozone</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -236,7 +256,6 @@
limitations under the License.
-->
-
<div class="panel panel-default">
<div class="panel-heading">Requirements</div>
<div class="panel-body">
@@ -244,67 +263,58 @@
<li>Working docker setup</li>
<li>AWS CLI (optional)</li>
</ul>
+
</div>
</div>
-
<h1 id="ozone-in-a-single-container">Ozone in a Single Container</h1>
-
<p>The easiest way to start up an all-in-one ozone container is to use the latest
docker image from docker hub:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker run -p <span style="color:#ae81ff">9878</span>:9878 -p <span style="color:#ae81ff">9876</span>:9876 apache/ozone</code></pre></div>
-<p>This command will pull down the ozone image from docker hub and start all
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run -p 9878:9878 -p 9876:9876 apache/ozone
+</span></span></code></pre></div><p>This command will pull down the ozone image from docker hub and start all
ozone services in a single container. <br>
This container will run the required metadata servers (Ozone Manager, Storage
Container Manager) one data node and the S3 compatible REST server
(S3 Gateway).</p>
-
<h1 id="local-multi-container-cluster">Local multi-container cluster</h1>
-
<p>If you would like to use a more realistic pseudo-cluster where each components
run in own containers, you can start it with a docker-compose file.</p>
-
<p>We have shipped a docker-compose and an enviorment file as part of the
container image that is uploaded to docker hub.</p>
-
<p>The following commands can be used to extract these files from the image in the docker hub.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker run apache/ozone cat docker-compose.yaml > docker-compose.yaml
-docker run apache/ozone cat docker-config > docker-config</code></pre></div>
-<p>Now you can start the cluster with docker-compose:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose up -d</code></pre></div>
-<p>If you need multiple datanodes, we can just scale it up:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"> docker-compose scale datanode<span style="color:#f92672">=</span><span style="color:#ae81ff">3</span></code></pre></div>
-<h1 id="running-s3-clients">Running S3 Clients</h1>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run apache/ozone cat docker-compose.yaml > docker-compose.yaml
+</span></span><span style="display:flex;"><span>docker run apache/ozone cat docker-config > docker-config
+</span></span></code></pre></div><p>Now you can start the cluster with docker-compose:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose up -d
+</span></span></code></pre></div><p>If you need multiple datanodes, we can just scale it up:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span> docker-compose scale datanode<span style="color:#f92672">=</span><span style="color:#ae81ff">3</span>
+</span></span></code></pre></div><h1 id="running-s3-clients">Running S3 Clients</h1>
<p>Once the cluster is booted up and ready, you can verify its status by
connecting to the SCM’s UI at <a href="http://localhost:9876">http://localhost:9876</a>.</p>
-
<p>The S3 gateway endpoint will be exposed at port 9878. You can use Ozone’s S3
support as if you are working against the real S3.</p>
-
<p>Here is how you create buckets from command line:</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3api --endpoint http://localhost:9878/ create-bucket --bucket<span style="color:#f92672">=</span>bucket1</code></pre></div>
-<p>Only notable difference in the above command line is the fact that you have
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws s3api --endpoint http://localhost:9878/ create-bucket --bucket<span style="color:#f92672">=</span>bucket1
+</span></span></code></pre></div><p>Only notable difference in the above command line is the fact that you have
to tell the <em>endpoint</em> address to the aws s3api command.</p>
-
<p>Now let us put a simple file into the S3 Bucket hosted by Ozone. We will
start by creating a temporary file that we can upload to Ozone via S3 support.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ls -1 > /tmp/testfile</code></pre></div>
-<p>This command creates a temporary file that
- we can upload to Ozone. The next command actually uploads to Ozone’s S3
- bucket using the standard aws s3 command line interface.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3 --endpoint http://localhost:9878 cp --storage-class REDUCED_REDUNDANCY /tmp/testfile s3://bucket1/testfile</code></pre></div>
-<p><div class="alert alert-info" role="alert">
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ls -1 > /tmp/testfile
+</span></span></code></pre></div><p>This command creates a temporary file that
+we can upload to Ozone. The next command actually uploads to Ozone’s S3
+bucket using the standard aws s3 command line interface.</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws s3 --endpoint http://localhost:9878 cp --storage-class REDUCED_REDUNDANCY /tmp/testfile s3://bucket1/testfile
+</span></span></code></pre></div><div class="alert alert-info" role="alert">
Note: REDUCED_REDUNDANCY is required for the single container ozone, since it
has a single datanode. </div>
We can now verify that file got uploaded by running the list command against
-our bucket.</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">aws s3 --endpoint http://localhost:9878 ls s3://bucket1/testfile</code></pre></div>
-<p><div class="alert alert-info" role="alert"> You can also check the internal
+our bucket.
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws s3 --endpoint http://localhost:9878 ls s3://bucket1/testfile
+</span></span></code></pre></div><div class="alert alert-info" role="alert"> You can also check the internal
bucket browser supported by Ozone S3 interface by clicking on the below link.
<br>
</div>
-<a href="http://localhost:9878/bucket1?browser">http://localhost:9878/bucket1?browser</a></p>
+http://localhost:9878/bucket1?browser
diff --git a/docs/0.5.0-beta/tools.html b/docs/0.5.0-beta/tools.html
index c2738d7..9e31b21 100644
--- a/docs/0.5.0-beta/tools.html
+++ b/docs/0.5.0-beta/tools.html
@@ -17,6 +17,28 @@
<link href="./css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -219,13 +241,9 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone has a set of command line tools that can be used to manage ozone.</p>
-
<p>All these commands are invoked via the <code>ozone</code> script.</p>
-
<p>Daemon commands:</p>
-
<ul>
<li><strong>scm</strong> - Storage Container Manager service, via daemon can be started
or stopped.</li>
@@ -234,17 +252,13 @@ or stopped.</li>
stopped.</li>
<li><strong>s3g</strong> -</li>
</ul>
-
<p>Client commands:</p>
-
<ul>
<li><strong>sh</strong> - Primary command line interface for ozone to manage volumes/buckets/keys.</li>
<li><strong>fs</strong> - Runs a command on ozone file system (similar to <code>hdfs dfs</code>)</li>
<li><strong>version</strong> - Prints the version of Ozone and HDDS.</li>
</ul>
-
<p>Admin commands:</p>
-
<ul>
<li><strong>classpath</strong> - Prints the class path needed to get the hadoop jar and the
required libraries.</li>
@@ -257,14 +271,11 @@ Container Manager.</li>
<li><strong>genconf</strong> - Generate minimally required ozone configs and output to
ozone-site.xml.</li>
</ul>
-
<p>Test tools:</p>
-
<ul>
<li><strong>freon</strong> - Runs the ozone load generator.</li>
<li><strong>genesis</strong> - Developer Only, Ozone micro-benchmark application.</li>
</ul>
-
<p>For more information see the following subpages:</p>
diff --git a/docs/0.5.0-beta/tools/auditparser.html b/docs/0.5.0-beta/tools/auditparser.html
index df74629..b886413 100644
--- a/docs/0.5.0-beta/tools/auditparser.html
+++ b/docs/0.5.0-beta/tools/auditparser.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -229,13 +251,10 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Audit Parser tool can be used for querying the ozone audit logs.
This tool creates a sqllite database at the specified path. If the database
already exists, it will avoid creating a database.</p>
-
<p>The database contains only one table called <code>audit</code> defined as:</p>
-
<p>CREATE TABLE IF NOT EXISTS audit (
datetime text,
level varchar(7),
@@ -247,24 +266,17 @@ params text,
result varchar(7),
exception text,
UNIQUE(datetime,level,logger,user,ip,op,params,result))</p>
-
<p>Usage:
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone auditparser <path to db file> <span style="color:#f92672">[</span>COMMAND<span style="color:#f92672">]</span> <span style="color:#f92672">[</span>PARAM<span style="color:#f92672">]</span></code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone auditparser <path to db file> <span style="color:#f92672">[</span>COMMAND<span style="color:#f92672">]</span> <span style="color:#f92672">[</span>PARAM<span style="color:#f92672">]</span></span></span></code></pre></div></p>
<p>To load an audit log to database:
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone auditparser <path to db file> load <path to audit log></code></pre></div>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone auditparser <path to db file> load <path to audit log></span></span></code></pre></div>
Load command creates the audit table described above.</p>
-
<p>To run a custom read-only query:
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone auditparser <path to db file> query <<span style="color:#66d9ef">select</span> query enclosed within double quotes></code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone auditparser <path to db file> query <<span style="color:#66d9ef">select</span> query enclosed within double quotes></span></span></code></pre></div></p>
<p>Audit Parser comes with a set of templates(most commonly used queries).</p>
-
<p>To run a template query:
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone auditparser <path to db file> template <templateName></code></pre></div></p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone auditparser <path to db file> template <templateName></span></span></code></pre></div></p>
<p>Following templates are available:</p>
-
<table>
<thead>
<tr>
@@ -273,24 +285,21 @@ Load command creates the audit table described above.</p>
<th>SQL</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>top5users</td>
<td>Top 5 users</td>
<td>select user,count(*) as total from audit group by user order by total DESC limit 5</td>
</tr>
-
<tr>
<td>top5cmds</td>
<td>Top 5 commands</td>
<td>select op,count(*) as total from audit group by op order by total DESC limit 5</td>
</tr>
-
<tr>
<td>top5activetimebyseconds</td>
<td>Top 5 active times, grouped by seconds</td>
-<td>select substr(datetime,1,charindex(‘,’,datetime)-1) as dt,count(*) as thecount from audit group by dt order by thecount DESC limit 5</td>
+<td>select substr(datetime,1,charindex(’,’,datetime)-1) as dt,count(*) as thecount from audit group by dt order by thecount DESC limit 5</td>
</tr>
</tbody>
</table>
diff --git a/docs/0.5.0-beta/tools/genconf.html b/docs/0.5.0-beta/tools/genconf.html
index ad28b04..bc81de8 100644
--- a/docs/0.5.0-beta/tools/genconf.html
+++ b/docs/0.5.0-beta/tools/genconf.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -229,10 +251,8 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Genconf tool generates a template ozone-site.xml file at the specified path.
This template file can be edited to replace with proper values.</p>
-
<p><code>ozone genconf <path></code></p>
diff --git a/docs/0.5.0-beta/tools/index.xml b/docs/0.5.0-beta/tools/index.xml
index 13e5d29..1a9a020 100644
--- a/docs/0.5.0-beta/tools/index.xml
+++ b/docs/0.5.0-beta/tools/index.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Tools on Ozone</title>
@@ -6,11 +6,7 @@
<description>Recent content in Tools on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
- <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/tools/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate><atom:link href="/tools/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Generate Configurations</title>
<link>/tools/genconf.html</link>
@@ -48,4 +44,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/tools/scmcli.html b/docs/0.5.0-beta/tools/scmcli.html
index 29631c9..f1511ee 100644
--- a/docs/0.5.0-beta/tools/scmcli.html
+++ b/docs/0.5.0-beta/tools/scmcli.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -229,9 +251,7 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>SCM is the block service for Ozone. It is also the workhorse for ozone. But user process never talks to SCM. However, being able to read the state of SCM is useful.</p>
-
<p>SCMCLI allows the developer to access SCM directly. Please note: Improper usage of this tool can destroy your cluster. Unless you know exactly what you are doing, Please do <em>not</em> use this tool. In other words, this is a developer only tool. We might even remove this command in future to prevent improper use.</p>
diff --git a/docs/0.5.0-beta/tools/testtools.html b/docs/0.5.0-beta/tools/testtools.html
index 2ca579a..32c509d 100644
--- a/docs/0.5.0-beta/tools/testtools.html
+++ b/docs/0.5.0-beta/tools/testtools.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -213,9 +235,7 @@
<div class="col-md-9">
<h1>Testing tools</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -231,66 +251,36 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Testing is one of the most important part during the development of a distributed system. We have the following type of test.</p>
-
<p>This page includes our existing test tool which are part of the Ozone source base.</p>
-
<p>Note: we have more tests (like TCP-DS, TCP-H tests via Spark or Hive) which are not included here because they use external tools only.</p>
-
<h2 id="unit-test">Unit test</h2>
-
<p>As every almost every java project we have the good old unit tests inside each of our projects.</p>
-
<h2 id="integration-test-junit">Integration test (JUnit)</h2>
-
<p>Traditional unit tests are supposed to test only one unit, but we also have higher level unit tests. They use <code>MiniOzoneCluster</code> which is a helper method to start real daemons (scm,om,datanodes) during the unit test.</p>
-
<p>From maven/java point of view they are just simple unit tests (JUnit library is used) but to separate them (and solve some dependency problems) we moved all of these tests to <code>hadoop-ozone/integration-test</code></p>
-
<h2 id="smoketest">Smoketest</h2>
-
<p>We use docker-compose based pseudo-cluster to run different configuration of Ozone. To be sure that the different configuration can be started we implemented <em>acceptance</em> tests with the help of <a href="https://robotframework.org/">https://robotframework.org/</a>.</p>
-
<p>The smoketests are available from the distribution (<code>./smoketest</code>) but the robot files defines only the tests: usually they start CLI and check the output.</p>
-
<p>To run the tests in different environmente (docker-compose, kubernetes) you need a definition to start the containers and execute the right tests in the right containers.</p>
-
<p>These definition of the tests are included in the <code>compose</code> directory (check <code>./compose/*/test.sh</code> or <code>./compose/test-all.sh</code>).</p>
-
<p>For example a simple way to test the distribution packege:</p>
-
-<pre><code>cd compose/ozonze
+<pre tabindex="0"><code>cd compose/ozonze
./test.sh
-</code></pre>
-
-<h2 id="blockade">Blockade</h2>
-
+</code></pre><h2 id="blockade">Blockade</h2>
<p><a href="https://github.com/worstcase/blockade">Blockade</a> is a tool to test network failures and partitions (it’s inspired by the legendary <a href="https://jepsen.io/analyses">Jepsen tests</a>).</p>
-
<p>Blockade tests are implemented with the help of tests and can be started from the <code>./blockade</code> directory of the distrubution.</p>
-
-<pre><code>cd blocakde
+<pre tabindex="0"><code>cd blocakde
pip install pytest==2.8.7,blockade
python -m pytest -s .
-</code></pre>
-
-<p>See the README in the blockade directory for more details.</p>
-
+</code></pre><p>See the README in the blockade directory for more details.</p>
<h2 id="minichaosozonecluster">MiniChaosOzoneCluster</h2>
-
<p>This is a way to get <a href="https://en.wikipedia.org/wiki/Chaos_engineering">chaos</a> in your machine. It can be started from the source code and a MiniOzoneCluster (which starts real daemons) will be started and killed randomly.</p>
-
<h2 id="freon">Freon</h2>
-
<p>Freon is a command line application which is included in the Ozone distribution. It’s a load generator which is used in our stress tests.</p>
-
<p>For example:</p>
-
-<pre><code>ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 --replicationType=RATIS --factor=THREE
-</code></pre>
-
-<pre><code>***************************************************
+<pre tabindex="0"><code>ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 --replicationType=RATIS --factor=THREE
+</code></pre><pre tabindex="0"><code>***************************************************
Status: Success
Git Base Revision: 48aae081e5afacbb3240657556b26c29e61830c3
Number of Volumes created: 10
@@ -305,17 +295,11 @@ Average Time spent in key write: 00:00:10,894
Total bytes written: 10240000
Total Execution time: 00:00:16,898
***********************
-</code></pre>
-
-<p>For more information check the <a href="https://hadoop.apache.org/ozone/docs/0.4.0-alpha/freon.html">documentation page</a></p>
-
+</code></pre><p>For more information check the <a href="https://hadoop.apache.org/ozone/docs/0.4.0-alpha/freon.html">documentation page</a></p>
<h2 id="genesis">Genesis</h2>
-
<p>Genesis is a microbenchmarking tool. It’s also included in the distribution (<code>ozone genesis</code>) but it doesn’t require real cluster. It measures different part of the code in an isolated way (eg. the code which saves the data to the local RocksDB based key value stores)</p>
-
<p>Example run:</p>
-
-<pre><code> ozone genesis -benchmark=BenchMarkRocksDbStore
+<pre tabindex="0"><code> ozone genesis -benchmark=BenchMarkRocksDbStore
# JMH version: 1.19
# VM version: JDK 11.0.1, VM 11.0.1+13-LTS
# VM invoker: /usr/lib/jvm/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/bin/java
@@ -394,12 +378,12 @@ Iteration 20: 196335.579 ops/s
-Result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test":
+Result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test":
167433.864 ?(99.9%) 43530.883 ops/s [Average]
(min, avg, max) = (41926.816, 167433.864, 223433.864), stdev = 50130.230
CI (99.9%): [123902.981, 210964.748] (assumes normal distribution)
-Secondary result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test:?stack":
+Secondary result "org.apache.hadoop.ozone.genesis.BenchMarkRocksDbStore.test:?stack":
Stack profiler:
....[Thread state distributions]....................................................................
@@ -435,7 +419,6 @@ BenchMarkRocksDbStore.test 4 8
BenchMarkRocksDbStore.test:?stack 4 8 4 512 5000 16 64 thrpt NaN ---
</code></pre>
-
</div>
diff --git a/docs/0.5.0-beta/zh/concept.html b/docs/0.5.0-beta/zh/concept.html
index 4fa5ea0..bef8311 100644
--- a/docs/0.5.0-beta/zh/concept.html
+++ b/docs/0.5.0-beta/zh/concept.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -177,7 +199,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">Ozone 架构 </h3>
diff --git a/docs/0.5.0-beta/zh/concept/datanodes.html b/docs/0.5.0-beta/zh/concept/datanodes.html
index 426e614..544ed83 100644
--- a/docs/0.5.0-beta/zh/concept/datanodes.html
+++ b/docs/0.5.0-beta/zh/concept/datanodes.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -185,7 +207,6 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>TODO: content translations</p>
diff --git a/docs/0.5.0-beta/zh/concept/hdds.html b/docs/0.5.0-beta/zh/concept/hdds.html
index df04b80..e1f27e7 100644
--- a/docs/0.5.0-beta/zh/concept/hdds.html
+++ b/docs/0.5.0-beta/zh/concept/hdds.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -185,14 +207,12 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Storage container manager provides multiple critical functions for the Ozone
cluster. SCM acts as the cluster manager, Certificate authority, Block
manager and the Replica manager.</p>
-
<div class="col-sm-6">
<div class="media">
@@ -215,7 +235,6 @@ SCM is in charge of creating an Ozone cluster. When an SCM is booted up via <kbd
-
<div class="col-sm-6">
<div class="media">
@@ -242,7 +261,6 @@ token infrastructure depends on this certificate infrastructure.
-
<div class="col-sm-6">
<div class="media">
@@ -267,7 +285,6 @@ read and write these blocks directly.
-
<div class="col-sm-6">
<div class="media">
diff --git a/docs/0.5.0-beta/zh/concept/index.xml b/docs/0.5.0-beta/zh/concept/index.xml
index d2daa84..bdb8fe2 100644
--- a/docs/0.5.0-beta/zh/concept/index.xml
+++ b/docs/0.5.0-beta/zh/concept/index.xml
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>概念 on Ozone</title>
@@ -6,11 +6,7 @@
<description>Recent content in 概念 on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-us</language>
- <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/zh/concept/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate><atom:link href="/zh/concept/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>概览</title>
<link>/zh/concept/overview.html</link>
@@ -48,4 +44,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/zh/concept/overview.html b/docs/0.5.0-beta/zh/concept/overview.html
index f0c3cc0..33f055e 100644
--- a/docs/0.5.0-beta/zh/concept/overview.html
+++ b/docs/0.5.0-beta/zh/concept/overview.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -169,9 +191,7 @@
<div class="col-md-9">
<h1>概览</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -187,29 +207,17 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone 是一个分布式、多副本的对象存储系统，并针对大数据场景进行了专门的优化。Ozone 主要围绕可扩展性进行设计，目标是十亿数量级以上的对象存储。</p>
-
-<p>Ozone 通过对命名空间与块空间的管理进行分离，大大增加了其可扩展性，其中命名空间由 <a href="../../zh/concept/ozonemanager.html">Ozone Manager </a>（OM）管理，块空间由 <a href="../../zh/concept/hdds.html">Storage Container Manager</a>（SCM）管理。</p>
-
+<p>Ozone 通过对命名空间与块空间的管理进行分离，大大增加了其可扩展性，其中命名空间由 <a href="../../zh/concept/ozonemanager.html">Ozone Manager </a>（OM）管理，块空间由 [Storage Container Manager] (/zh/concept/hdds.html)（SCM）管理。</p>
<p>Ozone 的管理由卷、桶和键组成。卷类似于个人主目录，只有管理员可以创建。</p>
-
<p>卷用来存储桶，用户可以在一个卷中创建任意数量的桶，桶中包含键，在 Ozone 中通过键来存储数据。</p>
-
<p>Ozone 的命名空间由存储卷组成，同时存储卷也用作存储账户管理。</p>
-
<p>下面的框图展示了 Ozone 的核心组件：</p>
-
-<p><img src="ozoneBlockDiagram.png" alt="架构图" /></p>
-
+<p><img src="ozoneBlockDiagram.png" alt="架构图"></p>
<p>Ozone Manager 管理命名空间，Storage Container Manager 管理底层的数据，而 Recon 是 Ozone 的管理接口。</p>
-
<h2 id="多种角度看-ozone">多种角度看 Ozone</h2>
-
-<p><img src="FunctionalOzone.png" alt="FunctionalOzone" /></p>
-
+<p><img src="FunctionalOzone.png" alt="FunctionalOzone"></p>
<p>任何分布式系统都可以从不同角度去观察。一种观察角度是，把 Ozone 看作是在分布式块存储（HDDS）之上加了一个命名空间服务（OM）。</p>
-
<p>另一种角度是把 Ozone 看作由几个不同的功能层组成：由 OM 和 SCM 组成的元数据管理层；由数据节点以及 SCM 构成的数据存储层；由 Ratis 提供的副本层，用来复制 OM 和 SCM 的元数据以及在修改数据节点上数据时保持数据一致性；Recon 是管理服务器，它负责和其它 Ozone 组件通信，并提供统一的管理 API 和界面。</p>
diff --git a/docs/0.5.0-beta/zh/concept/ozonemanager.html b/docs/0.5.0-beta/zh/concept/ozonemanager.html
index dc004ce..bf4a486 100644
--- a/docs/0.5.0-beta/zh/concept/ozonemanager.html
+++ b/docs/0.5.0-beta/zh/concept/ozonemanager.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -169,9 +191,7 @@
<div class="col-md-9">
<h1>Ozone Manager</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -187,49 +207,45 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone Manager（OM）管理 Ozone 的命名空间。</p>
-
<p>当向 Ozone 写入数据时，你需要向 OM 请求一个块，OM 会返回一个块并记录下相关信息。当你想要读取那个文件时，你也需要先通过 OM 获取那个块的地址。</p>
-
<p>OM 允许用户在卷和桶下管理键，卷和桶都是命名空间的一部分，也由 OM 管理。</p>
-
<p>每个卷都是 OM 下的一个独立命名空间的根，这一点和 HDFS 不同，HDFS 提供的是单个根目录的文件系统。</p>
-
<p>与 HDFS 中单根的树状结构相比，Ozone 的命名空间是卷的集合，或者可以看作是个森林，因此可以非常容易地部署多个 OM 来进行扩展。</p>
-
<h2 id="ozone-manager-元数据">Ozone Manager 元数据</h2>
-
<p>OM 维护了卷、桶和键的列表。它为每个用户维护卷的列表，为每个卷维护桶的列表，为每个桶维护键的列表。</p>
-
<p>OM 使用 Apache Ratis（Raft 协议的一种实现）来复制 OM 的状态，这为 Ozone 提供了高可用性保证。</p>
-
<h2 id="ozone-manager-和-storage-container-manager">Ozone Manager 和 Storage Container Manager</h2>
-
<p>为了方便理解 OM 和 SCM 之间的关系，我们来看看写入键和读取键的过程。</p>
-
<h3 id="写入键">写入键</h3>
-
<ul>
-<li><p>为了向 Ozone 中的某个卷下的某个桶的某个键写入数据，用户需要先向 OM 发起写请求，OM 会判断该用户是否有权限写入该键，如果权限许可，OM 分配一个块用于 Ozone 客户端数据写入。</p></li>
-
-<li><p>OM 通过 SCM 请求分配一个块（SCM 是数据节点的管理者），SCM 选择三个数据节点，分配新块并向 OM 返回块标识。</p></li>
-
-<li><p>OM 在自己的元数据中记录下块的信息，然后将块和块 token（带有向该块写数据的授权）返回给用户。</p></li>
-
-<li><p>用户使用块 token 证明自己有权限向该块写入数据，并向对应的数据节点写入数据。</p></li>
-
-<li><p>数据写入完成后，用户会更新该块在 OM 中的信息。</p></li>
+<li>
+<p>为了向 Ozone 中的某个卷下的某个桶的某个键写入数据，用户需要先向 OM 发起写请求，OM 会判断该用户是否有权限写入该键，如果权限许可，OM 分配一个块用于 Ozone 客户端数据写入。</p>
+</li>
+<li>
+<p>OM 通过 SCM 请求分配一个块（SCM 是数据节点的管理者），SCM 选择三个数据节点，分配新块并向 OM 返回块标识。</p>
+</li>
+<li>
+<p>OM 在自己的元数据中记录下块的信息，然后将块和块 token（带有向该块写数据的授权）返回给用户。</p>
+</li>
+<li>
+<p>用户使用块 token 证明自己有权限向该块写入数据，并向对应的数据节点写入数据。</p>
+</li>
+<li>
+<p>数据写入完成后，用户会更新该块在 OM 中的信息。</p>
+</li>
</ul>
-
<h3 id="读取键">读取键</h3>
-
<ul>
-<li><p>键读取相对比较简单，用户首先向 OM 请求该键的块列表。</p></li>
-
-<li><p>OM 返回块列表以及对应的块 token。</p></li>
-
-<li><p>用户连接数据节点，出示块 token，然后读取键数据。</p></li>
+<li>
+<p>键读取相对比较简单，用户首先向 OM 请求该键的块列表。</p>
+</li>
+<li>
+<p>OM 返回块列表以及对应的块 token。</p>
+</li>
+<li>
+<p>用户连接数据节点，出示块 token，然后读取键数据。</p>
+</li>
</ul>
diff --git a/docs/0.5.0-beta/zh/gdpr.html b/docs/0.5.0-beta/zh/gdpr.html
index 07e06c6..1525543 100644
--- a/docs/0.5.0-beta/zh/gdpr.html
+++ b/docs/0.5.0-beta/zh/gdpr.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -177,7 +199,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">GDPR 在 Ozone 中的应用 </h3>
@@ -188,11 +209,9 @@
</p>
</div>
</div>
-
<div class="alert alert-warning" role="alert">
如果你想详细了解 Ozone 的 GDPR 框架，请参考 <a href="https://issues.apache.org/jira/secure/attachment/12978992/Ozone%20GDPR%20Framework.pdf">Ozone GDPR 框架</a>。
</div>
-
<p>当你创建了一个符合 GDPR 规范的桶，在这个桶中创建的键都将自动符合 GDPR 规范。</p>
diff --git a/docs/0.5.0-beta/zh/gdpr/gdpr-in-ozone.html b/docs/0.5.0-beta/zh/gdpr/gdpr-in-ozone.html
index c884d2a..927ca94 100644
--- a/docs/0.5.0-beta/zh/gdpr/gdpr-in-ozone.html
+++ b/docs/0.5.0-beta/zh/gdpr/gdpr-in-ozone.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -185,17 +207,11 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>在 Ozone 中遵守 GDPR 规范非常简单，只需要在创建桶时指定 <code>--enforcegdpr=true</code> 或 <code>-g=true</code> 参数，这样创建出的桶都是符合 GDPR 规范的，当然，在桶中创建的键也都自动符合。</p>
-
<p>GDPR 只能在新桶上应用，对于已有的桶，要想使它符合 GDPR 规范，只能创建一个符合 GDPR 规范的新桶，然后把旧桶中的数据拷贝到新桶。</p>
-
<p>创建符合 GDPR 规范的桶的示例：</p>
-
<p><code>ozone sh bucket create --enforcegdpr=true /hive/jan</code></p>
-
<p><code>ozone sh bucket create -g=true /hive/jan</code></p>
-
<p>如果你想创建普通的桶，省略 <code>--enforcegdpr</code> 和 <code>-g</code> 参数即可。</p>
diff --git a/docs/0.5.0-beta/zh/gdpr/index.xml b/docs/0.5.0-beta/zh/gdpr/index.xml
index 52375d3..119a824 100644
--- a/docs/0.5.0-beta/zh/gdpr/index.xml
+++ b/docs/0.5.0-beta/zh/gdpr/index.xml
@@ -1,15 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>GDPR on Ozone</title>
<link>/zh/gdpr.html</link>
<description>Recent content in GDPR on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
-
- <atom:link href="/zh/gdpr/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/zh/gdpr/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Ozone 中的 GDPR</title>
<link>/zh/gdpr/gdpr-in-ozone.html</link>
@@ -20,4 +16,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/zh/index.html b/docs/0.5.0-beta/zh/index.html
index 854cc2a..f5a9d13 100644
--- a/docs/0.5.0-beta/zh/index.html
+++ b/docs/0.5.0-beta/zh/index.html
@@ -3,7 +3,7 @@
<!DOCTYPE html>
<html lang="en">
<head>
- <meta name="generator" content="Hugo 0.58.3" />
+ <meta name="generator" content="Hugo 0.108.0">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
@@ -18,6 +18,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -154,9 +176,7 @@
</div>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -172,22 +192,15 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<h1 id="apache-hadoop-ozone">Apache Hadoop Ozone</h1>
-
-<figure>
- <img src="../ozone-usage.png" width="60%"/>
+<figure><img src="../ozone-usage.png" width="60%"/>
</figure>
-
<p><em><em>Ozone 是 Hadoop 的分布式对象存储系统，具有易扩展和冗余存储的特点。<p>
Ozone 不仅能存储数十亿个不同大小的对象，还支持在容器化环境（比如 Kubernetes）中运行。</em></em> <p></p>
-
<p>Apache Spark、Hive 和 YARN 等应用无需任何修改即可使用 Ozone。Ozone 提供了 <a href="../zh/interface/javaapi.html">Java API</a>、<a href="../zh/interface/s3.html">S3 接口</a>和<a href="../zh/shell.html">命令行接口</a>，极大地方便了 Ozone
- 在不同应用场景下的的使用。</p>
-
+在不同应用场景下的的使用。</p>
<p>Ozone 的管理由卷、桶和键组成：</p>
-
<ul>
<li>卷的概念和用户账号类似，只有管理员可以创建和删除卷。</li>
<li>桶的概念和目录类似，用户可以在自己的卷下创建任意数量的桶，每个桶可以包含任意数量的键，但是不可以包含其它的桶。</li>
diff --git a/docs/0.5.0-beta/zh/index.xml b/docs/0.5.0-beta/zh/index.xml
index 23a6003..3005d8e 100644
--- a/docs/0.5.0-beta/zh/index.xml
+++ b/docs/0.5.0-beta/zh/index.xml
@@ -1,16 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>概述 on Ozone</title>
<link>/zh/</link>
<description>Recent content in 概述 on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
- <lastBuildDate>Tue, 10 Oct 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/zh/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/zh/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>概览</title>
<link>/zh/concept/overview.html</link>
@@ -125,7 +120,7 @@
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/zh/start/startfromdockerhub.html</guid>
- <description>Requirements docker AWS CLI（可选） 所有 Ozone 服务在单个容器 启动一个 all-in-one 的 ozone 容器最简单的方法就是使用 Docker Hub 最新的 docker 镜像：
+ <description>Requirements docker AWS CLI（可选） 所有 Ozone 服务在单个容器 启动一个 all-in-one 的 ozone 容器最简单的方法就是使用 Docker Hub 最新的 docker 镜像：
docker run -p 9878:9878 -p 9876:9876 apache/ozone 这个命令会从 Docker Hub 拉取 ozone 镜像并在一个容器中启动所有 ozone 服务，包括必要的元数据服务（Ozone Manager，Storage Container Manager）、一个数据节点和兼容 S3 的 REST 服务（S3 网关）。
Ozone 服务在多个独立的容器 如果你需要一个更类似生产环境的集群，使用 Ozone 发行包自带的 docker-compose 配置文件可以让 Ozone 服务组件在各自独立的容器中运行。
docker-compose 配置文件和一个 environment 文件已经包含在 Docker Hub 的镜像中。
@@ -140,10 +135,10 @@ docker run apache/ozone cat docker-compose.yaml &gt; docker-compose.yaml doc
<guid>/zh/start/onprem.html</guid>
<description>如果你想要有点挑战性，你可以在物理集群上安装 ozone。搭建一个 Ozone 集群需要了解它的各个组件，Ozone 既能和现有的 HDFS 集群并存运行，也可以独立运行。在这两种模式下，需要运行的 Ozone 组件是相同的。
-Ozone 组件 Ozone Manager - 管理 Ozone 命名空间的服务，负责所有对卷、桶和键的操作。 Storage Container Manager - Ozone 中块的管理者，Ozone Manager 从 SCM 请求块，然后用户向块写入数据。 Datanodes - Ozone 的 Datanode 代码既可以运行在 HDFS 的 Datanode 内，也可以独立部署成单独的进程。 搭建一个独立 Ozone 集群 将 ozone-&lt;version&gt; 安装包解压到目标目录，因为 Ozone 的 jar 包需要部署到集群的所有机器上，所以你需要在所有机器上进行此操作。
- Ozone 依赖名为 ozone-site.xml 的配置文件， 运行下面的命令可以在指定目录生成名为 ozone-site.xml 的配置文件模板，然后你可以将参数替换为合适的值。
- ozone genconf &lt;path&gt; 我们来看看生成的文件（ozone-site.xml）中都有哪些参数，以及它们是如何影响 ozone 的。当各个参数都配置了合适的值之后，需要把该文件拷贝到 ozone directory/etc/hadoop。
- ozone.</description>
+Ozone 组件 Ozone Manager - 管理 Ozone 命名空间的服务，负责所有对卷、桶和键的操作。 Storage Container Manager - Ozone 中块的管理者，Ozone Manager 从 SCM 请求块，然后用户向块写入数据。 Datanodes - Ozone 的 Datanode 代码既可以运行在 HDFS 的 Datanode 内，也可以独立部署成单独的进程。 搭建一个独立 Ozone 集群 将 ozone-&lt;version&gt; 安装包解压到目标目录，因为 Ozone 的 jar 包需要部署到集群的所有机器上，所以你需要在所有机器上进行此操作。
+Ozone 依赖名为 ozone-site.xml 的配置文件， 运行下面的命令可以在指定目录生成名为 ozone-site.xml 的配置文件模板，然后你可以将参数替换为合适的值。
+ozone genconf &lt;path&gt; 我们来看看生成的文件（ozone-site.xml）中都有哪些参数，以及它们是如何影响 ozone 的。当各个参数都配置了合适的值之后，需要把该文件拷贝到 ozone directory/etc/hadoop。
+ozone.metadata.dirs 管理员通过此参数指定元数据的存储位置，通常应该选择最快的磁盘（比如 SSD，如果节点上有的话），OM、SCM 和 Datanode 会将元数据写入此路径。这是个必需的参数，如果不配置它，Ozone 会启动失败。 示例如下：</description>
</item>
<item>
@@ -152,12 +147,12 @@ Ozone 组件 Ozone Manager - 管理 Ozone 命名空间的服务，负责所有
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/zh/start/minikube.html</guid>
- <description>Requirements minikube kubectl ozone 发行包中的 kubernetes/examples 文件夹下包含了多种用途的 kubernetes 部署资源文件，这些资源文件默认使用 Docker Hub 上的 apache/ozone 镜像。
+ <description>Requirements minikube kubectl ozone 发行包中的 kubernetes/examples 文件夹下包含了多种用途的 kubernetes 部署资源文件，这些资源文件默认使用 Docker Hub 上的 apache/ozone 镜像。
使用 minikube 资源集在 minikube 上进行部署：
-cd kubernetes/examples/minikube kubectl apply -f . 使用下面的命令检查结果：
-kubectl get pod 注意：kubernetes/exampls/minikube 资源集为 minikube 部署进行了如下优化：
- 即使你只有一个主机，也可以运行多个 Datanode（在实际的生产集群中，每个物理主机上通常只运行一个 Datanode） Ozone 通过不同的节点端口提供服务 访问服务 现在你可以访问 Ozone 的各个服务，minikube 资源集为每个 web 端点额外定义了一个 NodePort 服务，NodePort 服务可以通过指定端口从任意节点访问：
-kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE datanode ClusterIP None &lt;none&gt; &lt;none&gt; 27s kubernetes ClusterIP 10.</description>
+cd kubernetes/examples/minikube kubectl apply -f . 使用下面的命令检查结果：
+kubectl get pod 注意：kubernetes/exampls/minikube 资源集为 minikube 部署进行了如下优化：
+即使你只有一个主机，也可以运行多个 Datanode（在实际的生产集群中，每个物理主机上通常只运行一个 Datanode） Ozone 通过不同的节点端口提供服务 访问服务 现在你可以访问 Ozone 的各个服务，minikube 资源集为每个 web 端点额外定义了一个 NodePort 服务，NodePort 服务可以通过指定端口从任意节点访问：
+kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE datanode ClusterIP None &lt;none&gt; &lt;none&gt; 27s kubernetes ClusterIP 10.96.0.1 &lt;none&gt; 443/TCP 118m om ClusterIP None &lt;none&gt; 9874/TCP 27s om-public NodePort 10.</description>
</item>
<item>
@@ -166,12 +161,12 @@ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE datanode ClusterIP
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/zh/start/kubernetes.html</guid>
- <description>Requirements 可用的 kubernetes 集群（LoadBalancer 和 PersistentVolume 非必需） kubectl 由于 apache/ozone 镜像可以从 Docker Hub 获取到，K8s 上的部署过程和 Minikube 上的部署过程十分相似，唯一的区别是我们为 K8s 部署准备了专门的配置 文件（比如，我们可以在每个 K8s 节点上部署一个 Datanode）。
+ <description> Requirements 可用的 kubernetes 集群（LoadBalancer 和 PersistentVolume 非必需） kubectl 由于 apache/ozone 镜像可以从 Docker Hub 获取到，K8s 上的部署过程和 Minikube 上的部署过程十分相似，唯一的区别是我们为 K8s 部署准备了专门的配置 文件（比如，我们可以在每个 K8s 节点上部署一个 Datanode）。
ozone 安装包中的 kubernetes/examples 目录包含了为不同用例设计的 Kubernetes 部署资源文件。
使用 ozone 子目录进行部署：
-cd kubernetes/examples/ozone kubectl apply -f . 用下面的命令检查结果：
-kubectl get pod 访问 ozone 服务 现在你可以访问 ozone 的各个服务，默认情况下它们的端口并没有向外开放，不过你可以通过设置端口转发规则来开放外部访问：
-kubectl port-forward s3g-0 9878:9878 kubectl port-forward scm-0 9876:9876 </description>
+cd kubernetes/examples/ozone kubectl apply -f . 用下面的命令检查结果：
+kubectl get pod 访问 ozone 服务 现在你可以访问 ozone 的各个服务，默认情况下它们的端口并没有向外开放，不过你可以通过设置端口转发规则来开放外部访问：
+kubectl port-forward s3g-0 9878:9878 kubectl port-forward scm-0 9876:9876 </description>
</item>
<item>
@@ -180,10 +175,12 @@ kubectl port-forward s3g-0 9878:9878 kubectl port-forward scm-0 9876:9876 </des
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/zh/start/runningviadocker.html</guid>
- <description>Requirements docker 和 docker-compose 下载 Ozone 二进制压缩包并解压。
- 进入 docker compose 文件所在的目录，执行 docker-compose 命令，你的机器会启动一个运行在后台的 ozone 伪集群。
- cd compose/ozone/ docker-compose up -d 为了验证 Ozone 正常运行，我们可以登录到 Datanode 并运行 Ozone 的负载生成工具 _freon_。 exec datanode bash 命令会在 Datanode 上启动一个 bash，ozone freon 命令在 Datanode 所在的容器内执行，你随时可以通过 CTRL-C 退出 freon，命令行选项 rk 会让 freon 生成随机的键。
-docker-compose exec datanode bash ozone freon rk 你可以通过 http://localhost:9874/ 访问 OzoneManager UI 来查看服务端处理 freon 负载的情况，以及浏览 ozone 的配置。</description>
+ <description>Requirements docker 和 docker-compose 下载 Ozone 二进制压缩包并解压。
+进入 docker compose 文件所在的目录，执行 docker-compose 命令，你的机器会启动一个运行在后台的 ozone 伪集群。
+cd compose/ozone/ docker-compose up -d 为了验证 Ozone 正常运行，我们可以登录到 Datanode 并运行 Ozone 的负载生成工具 freon。 exec datanode bash 命令会在 Datanode 上启动一个 bash，ozone freon 命令在 Datanode 所在的容器内执行，你随时可以通过 CTRL-C 退出 freon，命令行选项 rk 会让 freon 生成随机的键。
+docker-compose exec datanode bash ozone freon rk 你可以通过 http://localhost:9874/ 访问 OzoneManager UI 来查看服务端处理 freon 负载的情况，以及浏览 ozone 的配置。
+恭喜，你成功运行了你的第一个 ozone 集群。
+关闭集群的命令为： docker-compose down</description>
</item>
<item>
@@ -192,7 +189,7 @@ docker-compose exec datanode bash ozone freon rk 你可以通过 http://localhos
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>/zh/start/fromsource.html</guid>
- <description>Requirements Java 1.8 Maven Protoc (2.5) 本文档是关于从源码构建 Ozone 的指南，如果你不打算亲自这么做，你大可放心地跳过本页。 如果你十分了解 Hadoop，并且熟悉 Apache 之道，那你应当知道 Apache 发行包的精髓在于源代码。
+ <description>Requirements Java 1.8 Maven Protoc (2.5) 本文档是关于从源码构建 Ozone 的指南，如果你不打算亲自这么做，你大可放心地跳过本页。 如果你十分了解 Hadoop，并且熟悉 Apache 之道，那你应当知道 Apache 发行包的精髓在于源代码。
从源码构建 ozone 只需要解压源码压缩包然后运行构建命令即可，下面这条命令假设你的机器上拥有构建 Hadoop 所需的所有环境，如果你需要构建 Hadoop 的指南，请查看 Apache Hadoop 网站。
mvn clean package -DskipTests=true 命令执行完成后，hadoop-ozone/dist/target 目录下会生成一个 ozone-&lt;version&gt;.tar.gz 文件。
你可以拷贝和使用这个压缩包来替代官方发行的二进制包。
@@ -202,4 +199,4 @@ cd smoketest ./test.sh --env ozone basic 验收测试会启动一个基于 docke
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/zh/interface.html b/docs/0.5.0-beta/zh/interface.html
index 522ce2e..edf2f06 100644
--- a/docs/0.5.0-beta/zh/interface.html
+++ b/docs/0.5.0-beta/zh/interface.html
@@ -17,6 +17,28 @@
<link href="../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -177,7 +199,6 @@
-->
-
<div class="jumbotron jumbotron-fluid">
<div class="container">
<h3 class="display-4">多协议支持 </h3>
diff --git a/docs/0.5.0-beta/zh/interface/index.xml b/docs/0.5.0-beta/zh/interface/index.xml
index 801c0b6..e7d3bde 100644
--- a/docs/0.5.0-beta/zh/interface/index.xml
+++ b/docs/0.5.0-beta/zh/interface/index.xml
@@ -1,16 +1,11 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
+<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>编程接口 on Ozone</title>
<link>/zh/interface.html</link>
<description>Recent content in 编程接口 on Ozone</description>
<generator>Hugo -- gohugo.io</generator>
- <language>en-us</language>
- <lastBuildDate>Thu, 14 Sep 2017 00:00:00 +0000</lastBuildDate>
-
- <atom:link href="/zh/interface/index.xml" rel="self" type="application/rss+xml" />
-
-
+ <language>en-us</language><atom:link href="/zh/interface/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Java API</title>
<link>/zh/interface/javaapi.html</link>
@@ -39,4 +34,4 @@
</item>
</channel>
-</rss>
\ No newline at end of file
+</rss>
diff --git a/docs/0.5.0-beta/zh/interface/javaapi.html b/docs/0.5.0-beta/zh/interface/javaapi.html
index cfd5d31..e630f33 100644
--- a/docs/0.5.0-beta/zh/interface/javaapi.html
+++ b/docs/0.5.0-beta/zh/interface/javaapi.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -169,9 +191,7 @@
<div class="col-md-9">
<h1>Java API</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -187,119 +207,94 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone 自带了支持 RPC 的客户端库，对于一般的应用场景也可以使用兼容 AWS S3 的 REST 接口替代 Ozone 客户端 API。</p>
-
<h2 id="创建-ozone-客户端">创建 Ozone 客户端</h2>
-
<p>ozone 客户端由客户端工厂创建，我们可以通过如下调用获得一个 RPC 客户端对象：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">OzoneClient <span style="color:#a6e22e">ozClient</span> <span style="color:#f92672">=</span> OzoneClientFactory.<span style="color:#a6e22e">getRpcClient</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>OzoneClient ozClient <span style="color:#f92672">=</span> OzoneClientFactory<span style="color:#f92672">.</span><span style="color:#a6e22e">getRpcClient</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>如果用户想要创建指定配置的客户端，可以这样调用：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">OzoneClient <span style="color:#a6e22e">ozClient</span> <span style="color:#f92672">=</span> OzoneClientFactory.<span style="color:#a6e22e">getClient</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>OzoneClient ozClient <span style="color:#f92672">=</span> OzoneClientFactory<span style="color:#f92672">.</span><span style="color:#a6e22e">getClient</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>这样就会返回一个合适的客户端对象。</p>
-
<h2 id="使用-ozone-客户端写数据">使用 Ozone 客户端写数据</h2>
-
<p>ozone 中的数据层次为卷、桶和键。卷是桶的集合，桶是键的集合，如果要向 ozone 写数据，你需要依次获得卷、桶和键。</p>
-
<h3 id="创建卷">创建卷</h3>
-
<p>有了客户端对象之后，我们需要获取 ObjectStore 的引用，获取方法为：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java">ObjectStore <span style="color:#a6e22e">objectStore</span> <span style="color:#f92672">=</span> ozClient.<span style="color:#a6e22e">getObjectStore</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span>ObjectStore objectStore <span style="color:#f92672">=</span> ozClient<span style="color:#f92672">.</span><span style="color:#a6e22e">getObjectStore</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>ObjectStore 对象表示了客户端连接的集群。</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// 让我们创建一个卷来存储资产数据
-</span><span style="color:#75715e">// 使用默认参数来创建资产卷
-</span><span style="color:#75715e"></span>objectStore.<span style="color:#a6e22e">createVolume</span>(<span style="color:#e6db74">"assets"</span>);
-
-<span style="color:#75715e">// 验证 assets 卷是否已创建
-</span><span style="color:#75715e"></span>OzoneVolume <span style="color:#a6e22e">assets</span> <span style="color:#f92672">=</span> objectStore.<span style="color:#a6e22e">getVolume</span>(<span style="color:#e6db74">"assets"</span>);</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// 让我们创建一个卷来存储资产数据
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// 使用默认参数来创建资产卷
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">createVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 验证 assets 卷是否已创建
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneVolume assets <span style="color:#f92672">=</span> objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">getVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span></span></span></code></pre></div>
<p>createVolume 方法也支持传入参数数组来指定卷的创建参数。</p>
-
<h3 id="创建桶">创建桶</h3>
-
<p>有了卷之后，你就可以在卷中创建桶。</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// 创建一个名为 videos 的桶
-</span><span style="color:#75715e"></span>assets.<span style="color:#a6e22e">createBucket</span>(<span style="color:#e6db74">"videos"</span>);
-OzoneBucket <span style="color:#a6e22e">video</span> <span style="color:#f92672">=</span> assets.<span style="color:#a6e22e">getBucket</span>(<span style="color:#e6db74">"videos"</span>);</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// 创建一个名为 videos 的桶
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>assets<span style="color:#f92672">.</span><span style="color:#a6e22e">createBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>OzoneBucket video <span style="color:#f92672">=</span> assets<span style="color:#f92672">.</span><span style="color:#a6e22e">getBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span></span></span></code></pre></div>
<p>此时我们有了一个可用的卷和一个桶，卷名为 <em>assets</em> ，桶名为 <em>videos</em> 。</p>
-
<p>接下来我们创建一个键。</p>
-
<h3 id="键的读写">键的读写</h3>
-
<p>通过桶对象可以进行键的读写，下面的代码从本地磁盘读取名为 intro.mp4 的视频文件，并把它存储到我们刚刚创建的 <em>video</em> 桶中。</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// 从文件读取数据，需要由用户提供读取函数
-</span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> [] videoData <span style="color:#f92672">=</span> readFile(<span style="color:#e6db74">"intro.mp4"</span>);
-
-<span style="color:#75715e">// 创建一个输出流并写数据
-</span><span style="color:#75715e"></span>OzoneOutputStream <span style="color:#a6e22e">videoStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">createKey</span>(<span style="color:#e6db74">"intro.mp4"</span>, 1048576);
-videoStream.<span style="color:#a6e22e">write</span>(videoData);
-
-<span style="color:#75715e">// 写操作完成之后关闭输出流
-</span><span style="color:#75715e"></span>videoStream.<span style="color:#a6e22e">close</span>();
-
-
-<span style="color:#75715e">// 我们可以使用同一个桶，通过创建输入流来读取刚刚写入的文件
-</span><span style="color:#75715e">// 先创建一个用来存储视频的字节数组
-</span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span>[] <span style="color:#a6e22e">data</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span>[(<span style="color:#66d9ef">int</span>)1048576];
-OzoneInputStream <span style="color:#a6e22e">introStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">readKey</span>(<span style="color:#e6db74">"intro.mp4"</span>);
-<span style="color:#75715e">// 读取 intro.mp4 到缓冲区中
-</span><span style="color:#75715e"></span>introStream.<span style="color:#a6e22e">read</span>(data);
-introStream.<span style="color:#a6e22e">close</span>();</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// 从文件读取数据，需要由用户提供读取函数
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> <span style="color:#f92672">[]</span> videoData <span style="color:#f92672">=</span> readFile<span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 创建一个输出流并写数据
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneOutputStream videoStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">createKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">,</span> <span style="color:#ae81ff">1048576</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">write</span><span style="color:#f92672">(</span>videoData<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 写操作完成之后关闭输出流
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 我们可以使用同一个桶，通过创建输入流来读取刚刚写入的文件
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// 先创建一个用来存储视频的字节数组
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span><span style="color:#f92672">[]</span> data <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span><span style="color:#f92672">[(</span><span style="color:#66d9ef">int</span><span style="color:#f92672">)</span><span style="color:#ae81ff">1048576</span><span style="color:#f92672">];</span>
+</span></span><span style="display:flex;"><span>OzoneInputStream introStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">readKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 读取 intro.mp4 到缓冲区中
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">read</span><span style="color:#f92672">(</span>data<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span></span></span></code></pre></div>
<p>下面是一个完整的代码示例，请注意代码中 close 函数的调用。</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#75715e">// 创建客户端对象
-</span><span style="color:#75715e"></span>OzoneClient <span style="color:#a6e22e">ozClient</span> <span style="color:#f92672">=</span> OzoneClientFactory.<span style="color:#a6e22e">getClient</span>();
-
-<span style="color:#75715e">// 通过客户端对象获取 ObjectStore 的引用
-</span><span style="color:#75715e"></span>ObjectStore <span style="color:#a6e22e">objectStore</span> <span style="color:#f92672">=</span> ozClient.<span style="color:#a6e22e">getObjectStore</span>();
-
-<span style="color:#75715e">// 创建用于存储数据的 assets 卷
-</span><span style="color:#75715e">// 此处创建的卷采用默认参数
-</span><span style="color:#75715e"></span>objectStore.<span style="color:#a6e22e">createVolume</span>(<span style="color:#e6db74">"assets"</span>);
-
-<span style="color:#75715e">// 验证卷是否已创建
-</span><span style="color:#75715e"></span>OzoneVolume <span style="color:#a6e22e">assets</span> <span style="color:#f92672">=</span> objectStore.<span style="color:#a6e22e">getVolume</span>(<span style="color:#e6db74">"assets"</span>);
-
-<span style="color:#75715e">// 创建名为 videos 的桶
-</span><span style="color:#75715e"></span>assets.<span style="color:#a6e22e">createBucket</span>(<span style="color:#e6db74">"videos"</span>);
-OzoneBucket <span style="color:#a6e22e">video</span> <span style="color:#f92672">=</span> assets.<span style="color:#a6e22e">getBucket</span>(<span style="color:#e6db74">"videos"</span>);
-
-<span style="color:#75715e">// 从文件中读取数据，需要用户提供此函数
-</span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> [] videoData <span style="color:#f92672">=</span> readFile(<span style="color:#e6db74">"intro.mp4"</span>);
-
-<span style="color:#75715e">// 创建输出流并写数据
-</span><span style="color:#75715e"></span>OzoneOutputStream <span style="color:#a6e22e">videoStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">createKey</span>(<span style="color:#e6db74">"intro.mp4"</span>, 1048576);
-videoStream.<span style="color:#a6e22e">write</span>(videoData);
-
-<span style="color:#75715e">// 写操作完成之后关闭输出流
-</span><span style="color:#75715e"></span>videoStream.<span style="color:#a6e22e">close</span>();
-
-
-<span style="color:#75715e">// 我们可以使用同一个桶，通过创建输入流来读取刚刚写入的文件
-</span><span style="color:#75715e">// 先创建一个用来存储视频的字节数组
-</span><span style="color:#75715e"></span>
-<span style="color:#66d9ef">byte</span>[] <span style="color:#a6e22e">data</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span>[(<span style="color:#66d9ef">int</span>)1048576];
-OzoneInputStream <span style="color:#a6e22e">introStream</span> <span style="color:#f92672">=</span> video.<span style="color:#a6e22e">readKey</span>(<span style="color:#e6db74">"intro.mp4"</span>);
-introStream.<span style="color:#a6e22e">read</span>(data);
-
-<span style="color:#75715e">// 读操作完成之后关闭输入流
-</span><span style="color:#75715e"></span>introStream.<span style="color:#a6e22e">close</span>();
-
-<span style="color:#75715e">// 关闭客户端
-</span><span style="color:#75715e"></span>ozClient.<span style="color:#a6e22e">close</span>();</code></pre></div>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-java" data-lang="java"><span style="display:flex;"><span><span style="color:#75715e">// 创建客户端对象
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneClient ozClient <span style="color:#f92672">=</span> OzoneClientFactory<span style="color:#f92672">.</span><span style="color:#a6e22e">getClient</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 通过客户端对象获取 ObjectStore 的引用
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>ObjectStore objectStore <span style="color:#f92672">=</span> ozClient<span style="color:#f92672">.</span><span style="color:#a6e22e">getObjectStore</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 创建用于存储数据的 assets 卷
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// 此处创建的卷采用默认参数
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">createVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 验证卷是否已创建
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneVolume assets <span style="color:#f92672">=</span> objectStore<span style="color:#f92672">.</span><span style="color:#a6e22e">getVolume</span><span style="color:#f92672">(</span><span style="color:#e6db74">"assets"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 创建名为 videos 的桶
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>assets<span style="color:#f92672">.</span><span style="color:#a6e22e">createBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>OzoneBucket video <span style="color:#f92672">=</span> assets<span style="color:#f92672">.</span><span style="color:#a6e22e">getBucket</span><span style="color:#f92672">(</span><span style="color:#e6db74">"videos"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 从文件中读取数据，需要用户提供此函数
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">byte</span> <span style="color:#f92672">[]</span> videoData <span style="color:#f92672">=</span> readFile<span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 创建输出流并写数据
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>OzoneOutputStream videoStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">createKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">,</span> <span style="color:#ae81ff">1048576</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">write</span><span style="color:#f92672">(</span>videoData<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 写操作完成之后关闭输出流
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>videoStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 我们可以使用同一个桶，通过创建输入流来读取刚刚写入的文件
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// 先创建一个用来存储视频的字节数组
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
+</span></span><span style="display:flex;"><span><span style="color:#66d9ef">byte</span><span style="color:#f92672">[]</span> data <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#66d9ef">byte</span><span style="color:#f92672">[(</span><span style="color:#66d9ef">int</span><span style="color:#f92672">)</span><span style="color:#ae81ff">1048576</span><span style="color:#f92672">];</span>
+</span></span><span style="display:flex;"><span>OzoneInputStream introStream <span style="color:#f92672">=</span> video<span style="color:#f92672">.</span><span style="color:#a6e22e">readKey</span><span style="color:#f92672">(</span><span style="color:#e6db74">"intro.mp4"</span><span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">read</span><span style="color:#f92672">(</span>data<span style="color:#f92672">);</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 读操作完成之后关闭输入流
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>introStream<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span>
+</span></span><span style="display:flex;"><span>
+</span></span><span style="display:flex;"><span><span style="color:#75715e">// 关闭客户端
+</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>ozClient<span style="color:#f92672">.</span><span style="color:#a6e22e">close</span><span style="color:#f92672">();</span></span></span></code></pre></div>
diff --git a/docs/0.5.0-beta/zh/interface/ozonefs.html b/docs/0.5.0-beta/zh/interface/ozonefs.html
index e253730..90f5619 100644
--- a/docs/0.5.0-beta/zh/interface/ozonefs.html
+++ b/docs/0.5.0-beta/zh/interface/ozonefs.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -169,9 +191,7 @@
<div class="col-md-9">
<h1>Ozone 文件系统</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -187,98 +207,61 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Hadoop 的文件系统接口兼容可以让任意像 Ozone 这样的存储后端轻松地整合进 Hadoop 生态系统，Ozone 文件系统就是一个兼容 Hadoop 的文件系统。</p>
-
<h2 id="搭建-ozone-文件系统">搭建 Ozone 文件系统</h2>
-
<p>要创建一个 ozone 文件系统，我们需要先为它选择一个用来存放数据的桶，这个桶会被用作 Ozone 文件系统的后端存储，所有的文件和目录都存储为这个桶中的键。</p>
-
<p>如果你还没有可用的卷和桶的话，请使用下面的命令创建：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ozone sh volume create /volume
-ozone sh bucket create /volume/bucket</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ozone sh volume create /volume
+</span></span><span style="display:flex;"><span>ozone sh bucket create /volume/bucket</span></span></code></pre></div>
<p>创建之后，请使用 <em>list volume</em> 或 <em>list bucket</em> 命令来确认桶已存在。</p>
-
<p>请在 core-site.xml 中添加以下条目：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.o3fs.impl<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzoneFileSystem<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span>
-<span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.AbstractFileSystem.o3fs.impl<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzFs<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span>
-<span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>fs.defaultFS<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>o3fs://bucket.volume<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.o3fs.impl<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzoneFileSystem<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.AbstractFileSystem.o3fs.impl<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>org.apache.hadoop.fs.ozone.OzFs<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>fs.defaultFS<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>o3fs://bucket.volume<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span></span></span></code></pre></div>
<p>这样会使指定的桶成为 HDFS 的 dfs 命令的默认文件系统，并且将其注册为了 o3fs 文件系统类型。</p>
-
<p>你还需要将 ozone-filesystem.jar 文件加入 classpath：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">export HADOOP_CLASSPATH<span style="color:#f92672">=</span>/opt/ozone/share/ozonefs/lib/hadoop-ozone-filesystem-lib-current*.jar:$HADOOP_CLASSPATH</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>export HADOOP_CLASSPATH<span style="color:#f92672">=</span>/opt/ozone/share/ozonefs/lib/hadoop-ozone-filesystem-lib-current*.jar:$HADOOP_CLASSPATH</span></span></code></pre></div>
<p>当配置了默认的文件系统之后，用户可以运行 ls、put、mkdir 等命令，比如：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls /</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls /</span></span></code></pre></div>
<p>或</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -mkdir /users</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -mkdir /users</span></span></code></pre></div>
<p>或者 put 命令。换句话说，所有像 Hive、Spark 和 Distcp 的程序都会在这个文件系统上工作。
请注意，在这个桶里使用 Ozone 文件系统以外的方法来进行键的创建和删除时，最终都会体现为 Ozone 文件系统中的目录和文件的创建和删除。</p>
-
<p>注意：桶名和卷名不可以包含句点。
此外，文件系统的 URI 可以由桶名和卷名后跟着 OM 主机的 FQDN 和一个可选的端口组成，比如，你可以同时指定主机和端口：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:5678/key</code></pre></div>
-
-<p>如果 URI 未指定端口，将会尝试从 <code>ozone.om.address</code> 配置中获取端口，如果 <code>ozone.om.address</code> 未配置，则使用默认端口 <code>9862</code>，比如，我们在 <code>ozone-site.xml
-</code> 中配置 <code>ozone.om.address</code> 如下：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"> <span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.om.address<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>0.0.0.0:6789<span style="color:#f92672"></value></span>
- <span style="color:#f92672"></property></span></code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:5678/key</span></span></code></pre></div>
+<p>如果 URI 未指定端口，将会尝试从 <code>ozone.om.address</code> 配置中获取端口，如果 <code>ozone.om.address</code> 未配置，则使用默认端口 <code>9862</code>，比如，我们在 <code>ozone-site.xml </code> 中配置 <code>ozone.om.address</code> 如下：</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span> <span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.om.address<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>0.0.0.0:6789<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"></property></span></span></span></code></pre></div>
<p>当我们运行下面的命令：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls o3fs://bucket.volume.om-host.example.com/key</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls o3fs://bucket.volume.om-host.example.com/key</span></span></code></pre></div>
<p>它其实等价于：</p>
-
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:6789/key</code></pre></div>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:6789/key</span></span></code></pre></div>
<p>注意：在这种情况下，<code>ozone.om.address</code> 配置中只有端口号会被用到，主机名是被忽略的。</p>
-
-<h2 id="兼容旧版本-hadoop-legacy-jar-和-basicozonefilesystem">兼容旧版本 Hadoop（Legacy jar 和 BasicOzoneFilesystem）</h2>
-
+<h2 id="兼容旧版本-hadooplegacy-jar-和-basicozonefilesystem">兼容旧版本 Hadoop（Legacy jar 和 BasicOzoneFilesystem）</h2>
<p>Ozone 文件系统的 jar 包有两种类型，它们都包含了所有的依赖：</p>
-
<ul>
<li>share/ozone/lib/hadoop-ozone-filesystem-lib-current-VERSION.jar</li>
<li>share/ozone/lib/hadoop-ozone-filesystem-lib-legacy-VERSION.jar</li>
</ul>
-
<p>第一种 jar 包包含了在一个版本兼容的 hadoop（hadoop 3.2）中使用 Ozone 文件系统需要的所有依赖。</p>
-
<p>第二种 jar 包将所有依赖单独放在一个内部的目录，并且这个目录下的类会用一个特殊的类加载器来加载这些类。通过这种方法，旧版本的 hadoop 就可以使用 hadoop-ozone-filesystem-lib-legacy.jar（比如hadoop 3.1、hadoop 2.7 或者 spark+hadoop 2.7）。</p>
-
<p>和依赖的 jar 包类似， OzoneFileSystem 也有两种实现。</p>
-
<p>对于 Hadoop 3.0 之后的版本，你应当使用 <code>org.apache.hadoop.fs.ozone.OzoneFileSystem</code>，它是兼容 Hadoop 文件系统 API 的完整实现。</p>
-
<p>对于 Hadoop 2.x 的版本，你应该使用基础版本 <code>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</code>，两者实现基本相同，但是不包含在 Hadoop 3.0 中引入的特性和依赖（比如文件系统统计信息、加密桶等）。</p>
-
<h3 id="总结">总结</h3>
-
<p>下表总结了各个版本 Hadoop 应当使用的 jar 包和文件系统实现：</p>
-
<table>
<thead>
<tr>
@@ -287,26 +270,22 @@ ozone sh bucket create /volume/bucket</code></pre></div>
<th>OzoneFileSystem 实现</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>3.2</td>
<td>filesystem-lib-current</td>
<td>org.apache.hadoop.fs.ozone.OzoneFileSystem</td>
</tr>
-
<tr>
<td>3.1</td>
<td>filesystem-lib-legacy</td>
<td>org.apache.hadoop.fs.ozone.OzoneFileSystem</td>
</tr>
-
<tr>
<td>2.9</td>
<td>filesystem-lib-legacy</td>
<td>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</td>
</tr>
-
<tr>
<td>2.7</td>
<td>filesystem-lib-legacy</td>
@@ -314,7 +293,6 @@ ozone sh bucket create /volume/bucket</code></pre></div>
</tr>
</tbody>
</table>
-
<p>由此可知，低版本的 Hadoop 可以使用 hadoop-ozone-filesystem-lib-legacy.jar（比如 hadoop 2.7 或者 spark+hadoop 2.7）。</p>
diff --git a/docs/0.5.0-beta/zh/interface/s3.html b/docs/0.5.0-beta/zh/interface/s3.html
index 7e73574..b6e6ca3 100644
--- a/docs/0.5.0-beta/zh/interface/s3.html
+++ b/docs/0.5.0-beta/zh/interface/s3.html
@@ -17,6 +17,28 @@
<link href="../../css/ozonedoc.css" rel="stylesheet">
+
+ <script>
+ var _paq = window._paq = window._paq || [];
+
+
+
+ _paq.push(['disableCookies']);
+
+
+ _paq.push(['trackPageView']);
+ _paq.push(['enableLinkTracking']);
+ (function() {
+ var u="//analytics.apache.org/";
+ _paq.push(['setTrackerUrl', u+'matomo.php']);
+ _paq.push(['setSiteId', '34']);
+ var d=document, g=d.createElement('script'),
+s=d.getElementsByTagName('script')[0];
+ g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
+ })();
+ </script>
+
+
</head>
@@ -169,9 +191,7 @@
<div class="col-md-9">
<h1>S3 协议接口</h1>
-
-
-<!---
+ <!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
@@ -187,38 +207,25 @@
See the License for the specific language governing permissions and
limitations under the License.
-->
-
<p>Ozone 提供了兼容 S3 的 REST 接口，你可以使用任何兼容 S3 的工具来操作 Ozone 的对象存储数据。</p>
-
<h2 id="起步">起步</h2>
-
<p>S3 网关是提供兼容 S3 的 API 的独立组件，它应当和普通的 Ozone 组件分别启动。</p>
-
<p>你可以使用发行包启动一个基于 docker 的集群，其中会包括 S3 网关。</p>
-
<p>进入 <code>compose/ozones3</code> 目录，启动服务：</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">docker-compose up -d</code></pre></div>
-<p>你可以通过 <code>http://localhost:9878</code> 访问 S3 网关。</p>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker-compose up -d
+</span></span></code></pre></div><p>你可以通过 <code>http://localhost:9878</code> 访问 S3 网关。</p>
<h2 id="url-规范">URL 规范</h2>
-
-<p>Ozone S3 网关既支持虚拟主机风格的 URL 作为桶地址（形如：<a href="http://bucketname.host:9878），也支持多级路径风格的">http://bucketname.host:9878），也支持多级路径风格的</a> URL 作为桶地址（形如 <a href="http://host:9878/bucketname）。">http://host:9878/bucketname）。</a></p>
-
+<p>Ozone S3 网关既支持虚拟主机风格的 URL 作为桶地址（形如：http://bucketname.host:9878），也支持多级路径风格的 URL 作为桶地址（形如 http://host:9878/bucketname）。</p>
<p>默认使用多级路径风格，如果要使用虚拟主机风格的 URL，在 <code>ozone-site.xml</code> 中设置你的主域名：</p>
-<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><property></span>
- <span style="color:#f92672"><name></span>ozone.s3g.domain.name<span style="color:#f92672"></name></span>
- <span style="color:#f92672"><value></span>s3g.internal<span style="color:#f92672"></value></span>
-<span style="color:#f92672"></property></span></code></pre></div>
-<h2 id="桶浏览器">桶浏览器</h2>
-
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#f92672"><property></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><name></span>ozone.s3g.domain.name<span style="color:#f92672"></name></span>
+</span></span><span style="display:flex;"><span> <span style="color:#f92672"><value></span>s3g.internal<span style="color:#f92672"></value></span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672"></property></span>
+</span></span></code></pre></div><h2 id="桶浏览器">桶浏览器</h2>
<p>在桶的 URL 中加上 <code>?browser=true</code>，就可以在浏览器中浏览桶。</p>
-
-<p>比如，<code>testbucket</code> 桶中的内容可以在浏览器中用 <a href="http://localhost:9878/testbucket?browser=true">http://localhost:9878/testbucket?browser=true</a> 来查看。</p>
-
+<p>比如，<code>testbucket</code> 桶中的内容可以在浏览器中用 http://localhost:9878/testbucket?browser=true 来查看。</p>
<h2 id="已实现的-rest-端点">已实现的 REST 端点</h2>
-
<p>S3 网关服务的操作：</p>
-
<table>
<thead>
<tr>
@@ -226,7 +233,6 @@
<th>状态</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>GET 服务</td>
@@ -234,9 +240,7 @@
</tr>
</tbody>
</table>
-
<p>桶操作：</p>
-
<table>
<thead>
<tr>
@@ -245,32 +249,27 @@
<th>备注</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>GET 桶（列举对象）第二版</td>
<td>已实现</td>
<td></td>
</tr>
-
<tr>
<td>HEAD 桶</td>
<td>已实现</td>
<td></td>
</tr>
-
<tr>
<td>DELETE 桶</td>
<td>已实现</td>
<td></td>
</tr>
-
<tr>
<td>PUT 桶 (创建桶)</td>
<td>已实现</td>
<td></td>
</tr>
-
<tr>
<td>Delete 多个对象 (POST)</td>
<td>已实现</td>
@@ -278,9 +277,7 @@
</tr>
</tbody>
</table>
-
<p>对象操作：</p>
-
<table>
<thead>
<tr>
@@ -289,32 +286,27 @@
<th>备注</th>
</tr>
</thead>
-
<tbody>
<tr>
<td>PUT 对象</td>
... 51979 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org