You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "kerneltime (via GitHub)" <gi...@apache.org> on 2025/07/10 01:05:09 UTC
[PR] HDDS-9857. Migrate core concepts documentation from legacy site. [ozone-site]
kerneltime opened a new pull request, #149:
URL: https://github.com/apache/ozone-site/pull/149
## What changes were proposed in this pull request?
This migration includes comprehensive documentation for:
- Architecture overview with detailed component explanations
- Complete namespace documentation (volumes, buckets, layouts)
- Data replication and storage container concepts
- Security documentation including Kerberos and ACL configurations
- All associated diagrams and images
The content has been reorganized to follow the new site structure and all internal links have been updated to work with the new paths.
Generated-by: Claude and Gemini
## What is the link to the Apache Jira?
HDDS-9857
## How was this patch tested?
Check off which of the following tests were done on this change. If additional testing was done, please elaborate here as well.
- [ ] The CI checks on my fork are passing
- [ ] I verified the rendered content using a local preview
- [ ] I manually verified the steps provided in this change work as described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9857. Migrate core concepts documentation [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #149:
URL: https://github.com/apache/ozone-site/pull/149#discussion_r2718528175
##########
docs/03-core-concepts/01-namespace/01-volumes/03-quotas.md:
##########
@@ -0,0 +1,32 @@
+---
+sidebar_label: Quotas
+---
+
+# Volume Quotas
+
+
+## Quota Management
+
+Quotas provide a mechanism to control resource utilization at the volume level:
+
+- **Space Quotas**: Limit the total amount of storage (in bytes) a volume can use
+- **Namespace Quotas**: Limit the total number of objects that can be stored within a volume
+- **Hierarchical Enforcement**: Volume quotas apply to the aggregate of all contained buckets
+- **Visibility**: Quota usage and limits can be monitored through the CLI and management interfaces
+- **Enforcement**: When a quota is reached, write operations fail with a descriptive error message
+
Review Comment:
We can add a few notes:
1. By default, the volume quota is unlimited.
2. the total quota of the buckets, cannot exceed the volume quota.
3. Volume quota has no effect if bucket quota is not set.
4. Volume having linked bucket do not consume space quota for keys within linked bucket. Linked bucket keys will consume space quota of source volume and source bucket.
##########
docs/03-core-concepts/01-namespace/01-volumes/03-quotas.md:
##########
@@ -0,0 +1,32 @@
+---
+sidebar_label: Quotas
+---
+
+# Volume Quotas
+
+
+## Quota Management
Review Comment:
Ozone defines two types of quota: namespace quota and storage space quota.
A namespace quota limits the number of objects in a volume.
A storage space quota limits the maximum used space of a volume.
##########
docs/03-core-concepts/01-namespace/01-volumes/03-quotas.md:
##########
@@ -0,0 +1,32 @@
+---
+sidebar_label: Quotas
+---
+
+# Volume Quotas
+
+
+## Quota Management
+
+Quotas provide a mechanism to control resource utilization at the volume level:
+
+- **Space Quotas**: Limit the total amount of storage (in bytes) a volume can use
+- **Namespace Quotas**: Limit the total number of objects that can be stored within a volume
+- **Hierarchical Enforcement**: Volume quotas apply to the aggregate of all contained buckets
+- **Visibility**: Quota usage and limits can be monitored through the CLI and management interfaces
+- **Enforcement**: When a quota is reached, write operations fail with a descriptive error message
+
+### Quota Commands
+
+```bash
+# Set both space (1TB) and namespace (1 million objects) quotas simultaneously
+ozone sh volume setquota --space=1TB --count=1000000 /marketing
+
+# Check current quota settings and usage
+ozone sh volume info /marketing
+
+# Clear the space quota (set space limit to unlimited)
+ozone sh volume clrquota --space /marketing
+
+# Clear the namespace quota (set object count limit to unlimited)
+ozone sh volume clrquota --count /marketing
Review Comment:
```suggestion
ozone sh volume clrquota --namespace-quota /marketing
```
##########
docs/03-core-concepts/01-namespace/01-volumes/03-quotas.md:
##########
@@ -0,0 +1,32 @@
+---
+sidebar_label: Quotas
+---
+
+# Volume Quotas
+
+
+## Quota Management
+
+Quotas provide a mechanism to control resource utilization at the volume level:
+
+- **Space Quotas**: Limit the total amount of storage (in bytes) a volume can use
+- **Namespace Quotas**: Limit the total number of objects that can be stored within a volume
+- **Hierarchical Enforcement**: Volume quotas apply to the aggregate of all contained buckets
+- **Visibility**: Quota usage and limits can be monitored through the CLI and management interfaces
+- **Enforcement**: When a quota is reached, write operations fail with a descriptive error message
+
+### Quota Commands
+
+```bash
+# Set both space (1TB) and namespace (1 million objects) quotas simultaneously
+ozone sh volume setquota --space=1TB --count=1000000 /marketing
+
+# Check current quota settings and usage
+ozone sh volume info /marketing
+
+# Clear the space quota (set space limit to unlimited)
+ozone sh volume clrquota --space /marketing
Review Comment:
```suggestion
ozone sh volume clrquota --space-quota /marketing
```
##########
docs/03-core-concepts/01-namespace/01-volumes/03-quotas.md:
##########
@@ -0,0 +1,32 @@
+---
+sidebar_label: Quotas
+---
+
+# Volume Quotas
+
+
+## Quota Management
+
+Quotas provide a mechanism to control resource utilization at the volume level:
+
+- **Space Quotas**: Limit the total amount of storage (in bytes) a volume can use
+- **Namespace Quotas**: Limit the total number of objects that can be stored within a volume
+- **Hierarchical Enforcement**: Volume quotas apply to the aggregate of all contained buckets
+- **Visibility**: Quota usage and limits can be monitored through the CLI and management interfaces
+- **Enforcement**: When a quota is reached, write operations fail with a descriptive error message
+
Review Comment:
We can add a few notes:
1. By default, the volume quota is unlimited.
2. the total quota of the buckets, cannot exceed the volume quota.
3. Volume quota has no effect if bucket quota is not set.
4. Volume having linked bucket do not consume space quota for keys within linked bucket. Linked bucket keys will consume space quota of source volume and source bucket.
5. files in trash still occupies storage space and namespace. They are only decremented when trash is purged.
##########
docs/03-core-concepts/01-namespace/01-volumes/03-quotas.md:
##########
@@ -0,0 +1,32 @@
+---
+sidebar_label: Quotas
+---
+
+# Volume Quotas
+
+
+## Quota Management
+
+Quotas provide a mechanism to control resource utilization at the volume level:
+
+- **Space Quotas**: Limit the total amount of storage (in bytes) a volume can use
+- **Namespace Quotas**: Limit the total number of objects that can be stored within a volume
+- **Hierarchical Enforcement**: Volume quotas apply to the aggregate of all contained buckets
+- **Visibility**: Quota usage and limits can be monitored through the CLI and management interfaces
+- **Enforcement**: When a quota is reached, write operations fail with a descriptive error message
+
+### Quota Commands
+
+```bash
+# Set both space (1TB) and namespace (1 million objects) quotas simultaneously
+ozone sh volume setquota --space=1TB --count=1000000 /marketing
Review Comment:
```suggestion
bin/ozone sh volume setquota --namespace-quota 1000000 /marketing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9857. Migrate core concepts documentation [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #149:
URL: https://github.com/apache/ozone-site/pull/149#issuecomment-3054847732
@kerneltime please resolve conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9857. Migrate core concepts documentation [ozone-site]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #149:
URL: https://github.com/apache/ozone-site/pull/149#issuecomment-3872471440
@jojochuang is there a Jira or existing doc for end users to authenticate with Kerberos/SPNEGO on an existing secure cluster? They should not have to read the admin guide for this information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9857. Migrate core concepts documentation [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #149:
URL: https://github.com/apache/ozone-site/pull/149#issuecomment-3872999836
placeholder jira: https://issues.apache.org/jira/browse/HDDS-14466
Also this section in System Internals touches upon using curl with SPNEGO. Can move to Core Concepts. https://ozone-site-v2.staged.apache.org/docs/system-internals/security/kerberos#spnego-for-http-access
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9857. Migrate core concepts documentation [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang closed pull request #149: HDDS-9857. Migrate core concepts documentation
URL: https://github.com/apache/ozone-site/pull/149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9857. Migrate core concepts documentation from legacy site. [ozone-site]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #149:
URL: https://github.com/apache/ozone-site/pull/149#discussion_r2196283544
##########
docs/03-core-concepts/01-architecture.md:
##########
@@ -1,5 +1,327 @@
# Architecture
-**TODO:** File a subtask under [HDDS-9857](https://issues.apache.org/jira/browse/HDDS-9857) and complete this page or section.
+Apache Ozone is a scalable, distributed, and highly available object store designed to handle billions of objects of any size. This document provides an overview of Ozone's architecture, including its core components, data organization, and operational concepts.
-A high level overview of Ozone's components and what they do.
+## Ozone Namespace
+
+Ozone organizes data in a hierarchical namespace consisting of three levels:
+
+### Volumes
+
+Volumes are the top-level entities in the Ozone namespace, conceptually similar to filesystems in traditional storage systems. They typically represent:
+
+- Multi-tenant boundaries
+- Organizational divisions
+- Project groupings
+
+Volumes provide isolation and can have their own admins and quota limits.
+
+### Buckets
+
+Buckets exist within volumes and act as containers for objects (keys). Each bucket can be configured with specific properties:
+
+- Storage type and replication factor
+- Encryption settings
+- Access control policies
+- Quota limits
+
+Buckets are analogous to directories in a filesystem or buckets in cloud object stores.
+
+### Keys (Objects)
+
+Keys are the actual data objects stored in buckets. They can be:
+
+- Files of any size
+- Binary data
+- Named using a path-like structure depending on bucket layout
+
+For more details about Ozone's namespace, see the namespace documentation.
+
+### Ozone Bucket Types
+
+Ozone supports two distinct bucket layouts, each optimized for different use cases:
+
+#### Object Store Layout
+
+The Object Store layout (OBS) works like traditional object stores with a flat namespace:
+
+- Objects are stored with their full key path
+- No concept of directories (though path-like naming is supported)
+- Optimized for object storage workloads
+- Compatible with S3-style access patterns
+
+#### File System Optimized Layout
+
+The File System Optimized layout (FSO) provides hierarchical directory structure:
+
+- Directories are first-class entities
+- Supports efficient directory operations (listing, renaming)
+- Includes filesystem features like trash
+- Better performance for filesystem-style workloads
+- Default layout type
+
+The bucket layout determines how data is organized and accessed within a bucket.
+
+
+
+## Core Components
+
+Ozone has a modular architecture with several key components that work together to provide a scalable and reliable storage system.
+
+### Ozone Manager (OM)
+
+The Ozone Manager is the metadata server that manages the namespace:
+
+- Handles all volume, bucket, and key operations
+- Maintains metadata in RocksDB
+- Allocates blocks for data storage
+- Manages access control
+- Supports HA deployment with Ratis consensus protocol
+
+The OM is the entry point for all namespace operations. It tracks which objects exist and where they are stored.
+
+### Storage Container Manager (SCM)
+
+The Storage Container Manager orchestrates the container lifecycle and coordinates datanodes:
+
+- Manages container creation and allocation
+- Tracks datanode status and health
+- Handles container replication and EC
+- Issues block allocation requests
+- Coordinates container balancing
+- Supports HA deployment with Ratis
+
+SCM is the control plane for container management.
+
+### Datanode
+
+Datanodes are the workhorses that store the actual data:
+
+- Store data in containers on local disks
+- Serve read and write requests
+- Report container statistics to SCM
+- Participate in replication pipelines
+- Handle data integrity checks
+
+Each datanode manages a set of containers and serves read/write requests from clients.
+
+### Recon
+
+Recon is the analytics and monitoring component:
+
+- Collects and aggregates metrics
+- Provides a web UI for monitoring
+- Offers a consolidated view of the cluster
+- Helps identify issues and bottlenecks
+- Syncs data from OM, SCM, and Datanodes
+
+Recon is an optional but recommended component for operational visibility.
+
+### S3 Gateway
+
+The S3 Gateway provides S3-compatible API access:
+
+- Implements S3 REST API
+- Translates S3 operations to Ozone operations
+- Supports most S3 features and SDKs
+- Provides authentication and authorization
+
+The S3 Gateway enables applications built for S3 to work with Ozone.
+
+### HttpFS
+
+HttpFS provides a REST interface compatible with WebHDFS:
+
+- Enables HTTP access to Ozone
+- Compatible with HDFS clients
+- Supports read/write operations
+- Facilitates integration with web applications
+
+HttpFS allows web applications to interact with Ozone using familiar APIs.
+
+### Ozone Client
+
+The Ozone client is the software component that enables applications to interact with the Ozone storage system:
+
+- Provides Java libraries for programmatic access
+- Handles communication with OM for namespace operations
+- Manages direct data transfer with datanodes
+- Implements client-side caching for improved performance
+- Offers pluggable interfaces for different protocols (S3, OFS)
+- Handles authentication and token management
+
+The client library abstracts away the complexity of the distributed system, providing applications with a simple, consistent interface to Ozone storage.
+
+### Component Interactions
+
+The components of Ozone interact in well-defined patterns for different operations:
+
+
+
+#### Write Path Sequence
+
+The typical write sequence follows these steps:
+
+1. **Namespace Operations**: The client contacts the Ozone Manager to create or locate the key in the namespace
+2. **Block Allocation**: The Ozone Manager requests blocks from the Storage Container Manager
+3. **Data Transfer**: The client directly writes data to the selected Datanodes according to the replication pipeline
+4. **Key Commit**: After successful data transfer, the client commits the key to the Ozone Manager
+
+#### Read Path Sequence
+
+For reads, the process is simpler:
+
+1. The client requests key location information from the Ozone Manager
+2. Using the block location information, the client reads data directly from Datanodes
+3. In case of failures, the client retries with alternative replicas
+
+#### Monitoring and Management
+
+
+The Recon service continuously:
+- Collects metrics from the Ozone Manager, Storage Container Manager, and Datanodes
+- Provides consolidated views of system health and performance
+- Facilitates troubleshooting and management
+
+## Ozone Internals
+
+Understanding Ozone's internal structure helps to grasp how data is organized and protected.
+
+### Containers
+
+Containers are the fundamental storage units in Ozone:
+
+- Fixed-size (typically 5GB) units of storage
+- Managed by the Storage Container Manager (SCM)
+- Replicated or erasure-coded across datanodes
+- Contain multiple blocks
+- Include metadata and chunk files
+
+Containers are self-contained units that include all necessary metadata and data. They are the unit of replication in Ozone.
+
+### Blocks
+
+Blocks are logical units of data within containers:
+
+- Represent portions of objects/keys
+- Created when clients write data
+- Referenced by object metadata
+- Allocated by the Ozone Manager
+- Secured with block tokens
+
+When a client writes data, the OM allocates blocks from SCM, and the client writes data to these blocks through datanodes.
+
+### Chunks
+
+Chunks are the physical data units stored on disk:
+
+- Fixed-size portions of blocks
+- Written sequentially in container data files
+- Checksummed for data integrity
+- Optimized for disk I/O
+
+Chunks are the smallest units of data stored on disk and include checksums for integrity verification.
+
+
+
+### Replicated Containers
+
+Ozone provides durability through container replication:
+
+- Default replication factor is 3
+- Uses Ratis (Raft) consensus protocol
+- Synchronously replicates data across datanodes
+- Provides strong consistency guarantees
+- Handles node failures transparently
+
+Replicated containers ensure data durability by storing multiple copies of each container across different datanodes.
+
+
+
+### Erasure Encoded Containers
+
+Erasure coding provides space-efficient durability:
+
+- Splits data across multiple datanodes with parity
+- Supports various coding schemes (e.g., RS-3-2-1024k)
+- Reduces storage overhead compared to replication
+- Trades some performance for storage efficiency
+- Suitable for cold data storage
+
+Erasure coding allows for data durability with less storage overhead than full replication.
+
+
+
+### Pipelines
+
+Pipelines are groups of datanodes that work together to store data:
+
+- Managed by SCM
+- Consist of multiple datanodes
+- Handle write operations as a unit
+- Support different replication strategies
+
+For detailed information, see [Write Pipelines](http://localhost:3001/docs/core-concepts/replication/write-pipelines).
Review Comment:
This link points to a localhost address, which will break in production. Update it to a relative path like `./02-data-replication/02-write-pipelines.md` or a proper URL within the site.
```suggestion
For detailed information, see [Write Pipelines](./02-data-replication/02-write-pipelines.md).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org