You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/07/09 17:12:58 UTC
[PR] Add S3-compat and S3 conditional request blog [ozone-site]
peterxcli opened a new pull request, #488:
URL: https://github.com/apache/ozone-site/pull/488
## What changes were proposed in this pull request?
Add S3-compat and S3 conditional request blog
## What is the link to the Apache Jira?
1. Please create an issue in ASF Jira before opening a pull request.
2. Start the title of the pull request with the corresponding Jira issue number (e.g. HDDS-XXXX. Fix a typo in YYY).
3. Add the `website` and/or `documentation` labels to your Jira if applicable.
4. Replace this section with the link to the Apache Jira.
## How was this patch tested?
<img width="1020" height="1214" alt="image" src="https://github.com/user-attachments/assets/9719ca6d-12f8-458b-b170-1bc971092986" />
<img width="797" height="731" alt="image" src="https://github.com/user-attachments/assets/2875b648-23bd-4f4a-95e7-e0acb650aa04" />
<img width="855" height="744" alt="image" src="https://github.com/user-attachments/assets/679e0f98-228a-43fa-ab78-481bf1988227" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Add S3-compat and S3 conditional request blog [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #488:
URL: https://github.com/apache/ozone-site/pull/488#discussion_r3574376780
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
+
+1. First, we originally used one JSON file to represent one run result. Each latest run was about 0.236 MiB. This sounds small, but the problem is that we did not lazy load these files at the beginning. Every time the page refreshed, even if some files might already be cached by the browser, the TypeScript data structure still ended up holding too much content from too many files. That is very memory-consuming.
+
+2. Second, the JSON data had a ton of room to compress. The data shape is very repetitive: `run_id`, `suite_key`, `status`, feature names, source paths, class names, test names, and similar object keys appear again and again. JSON stores all of that as repeated text and nested objects. Even if HTTP compression helps over the network, the browser still has to parse it into JavaScript objects, and Git still stores changed blobs. This kind of data is much closer to a table than a document, so a columnar format makes more sense.
+
+3. Finally, the search index data was not normalized correctly. The index repeated run metadata and case metadata too much, and it grew quickly. For Git history, this is especially bad because every new run changes a large generated file again.
+
+So then I basically applied three optimizations: lazy loading, Parquet compression, and search/index normalization.
+
+1. **Lazy loading.** The page now loads the light catalog/index first. Full run details are fetched only when you open the latest run or expand an archived run. Search index loading is also delayed until you actually search, and the browser can keep the FlexSearch index in IndexedDB. Logs are fetched only when you open a log modal. Source snippets are fetched only when you open a case. This makes the first page load much cheaper.
+2. **Compress data as Parquet.** Instead of treating every run as one large JSON document, the report data is split into Parquet tables: catalog runs, suites, features, per-run metadata, per-run cases, search rows, log files, and log lines. The Parquet writer uses Zstd compression and dictionary encoding, which is a much better fit for repeated values like suite names, statuses, feature tags, and source paths. The frontend then queries these Parquet files in the browser through DuckDB-Wasm.
+3. **Index file optimization.** The search rows now contain normalized fields and a short `detail_preview`, while the full case detail can be hydrated from the per-run case Parquet file when needed. For JSON fallback, the index and search data are partitioned into smaller shards. For the normal Parquet path, the global search index is a compressed Parquet file. The search index also has a stable `index_id`, so the browser can reuse the IndexedDB cache when the data has not changed.
+
+Result is crazy:
+
+1. Parquet data now:
+ 1. search rows: 7.64 MiB, 33,876 rows, growing about 0.142-0.147 MiB/run
+ 2. run case/detail data excluding search: 4.92 MiB across 52 runs, growing about 0.093 MiB/run
+ 3. catalog/index data: 0.03 MiB now, growing less than 1 KiB/run
+ 4. total Pages data: 12.59 MiB across 52 runs
+ 5. latest run data: 246,696 bytes \= 0.235 MiB
+2. Projection at the current rate:
+ 1. about \+0.236 MiB/run
+ 2. about \+7.1 MiB/month
+ 3. about \+86 MiB/year
+2. Compared with the JSON baseline:
+ 1. current data size: 149.11 MiB \-\> 12.59 MiB, about 11.8x smaller
+ 2. yearly growth: \~1.0 GiB/year \-\> \~86 MiB/year, about 12x slower
+ 3. search data: 107.09 MiB \-\> 7.64 MiB, about 14x smaller
+
+## Current Effect
+
+Ozone has already improved since the dashboard was released.
+
+
+I don’t want to overclaim that every improvement below was caused only by this dashboard, but I do think the dashboard changed the workflow. It gives us concrete failing cases, links that can be pasted into PRs, and an easy way to check whether the gap is still there after a patch.
Review Comment:
or "I don’t want to take credit for every improvement ..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Add S3-compat and S3 conditional request blog [ozone-site]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #488:
URL: https://github.com/apache/ozone-site/pull/488#issuecomment-4962622909
cc @jojochuang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Add S3-compat and S3 conditional request blog [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #488:
URL: https://github.com/apache/ozone-site/pull/488#discussion_r3574364707
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
+
+AWS also noted that conditional requests [let customers remove workaround code and simplify their systems](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=When%20we%20moved%20S3%20to,similar%20reaction). The same storage-level feature also [powers S3 Tables](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=they%20involve%20a%20complex,storage%2Dlevel%20features), which manages tabular data on S3.
+
+Common conditional request patterns include:
+
+- **Conditional `PUT`**
+ - Use `If-None-Match: *` to create an object only if it does not already exist.
+ - Use `If-Match: <etag>` to overwrite an object only if the current ETag is still the one the client observed.
+ - Use case: metadata commits, manifest updates, distributed lock files, and create-only log chunks.
+- **Conditional `GET`**
+ - Use `If-None-Match: <etag>` or date-based conditions to download the object only when it has changed.
+ - Use case: cache validation, dynamic model reloading, config refresh, and avoiding repeated downloads of large objects.
+- **Conditional `HEAD`**
+ - Same idea as conditional `GET`, but metadata-only.
+ - Use case: checking whether an object is still current without fetching the body.
+- **Conditional `POST` / `CompleteMultipartUpload`**
+ - Multipart upload writes data in parts, but the destination object becomes visible only when `CompleteMultipartUpload` succeeds.
+ - Conditional completion lets the client say: “complete this large object only if the destination still does not exist” or “only if the destination still matches this ETag.”
+ - Use case: large object writes where the final commit still needs create-only or compare-and-swap semantics.
+- **Conditional `COPY`**
+ - Source-side conditions make sure the source object is the version the client expects.
+ - Destination-side conditions make sure the copy does not overwrite unexpected data.
+ - Use case: snapshot promotion, clone/copy workflows, and metadata copy where both source and destination need validation.
+- **Conditional `DELETE`**
+ - Use `If-Match: <etag>` to delete only if the object is still unchanged.
+ - Use `If-Match: *` to delete only if the object exists.
+ - Use case: preventing a client from deleting an object that another writer has already replaced.
+
+For more advanced usage, let’s look at how modern systems use these APIs.
+
+## Application use cases
+
+So who exactly can use this?
+
+### Dynamic AI model reloading
+
+Advanced inference servers often cache massive model weights on local NVMe drives. To keep these models updated without constantly re-downloading them every few minutes, they can use S3 conditional requests. For example, the inference server can send a `GET` request with `If-None-Match` and the ETag of the model it already has.
+
+If the ETag still matches, Ozone can return `304 Not Modified`, so the server keeps using the local cached model and skips the download. If the ETag does not match, the model has been updated, so the server downloads the newest model and stores the new ETag for the next check. Thanks to [@amaliujia](https://github.com/amaliujia) for the example.
+
+
+
+*Figure 1. An inference server uses `If-None-Match` to avoid re-downloading unchanged model weights.*
+
+### Turbopuffer
+
+Turbopuffer’s object-storage queue is a great example of using a small metadata object to coordinate larger data work. In the simple version, a pusher reads `queue.json`, appends a job, and writes it back with CAS. A worker does the same thing to claim the next job. The CAS write only succeeds if `queue.json` has not changed since the client read it; otherwise the client reads the new queue and retries.
+
+This design is simple, but many clients can contend on one queue object. Turbopuffer improves it by batching updates with group commit, then moving the object-storage interaction behind a stateless broker. Clients talk to the broker, the broker performs one group-commit loop, and it only acknowledges work after the updated queue is durably committed.
+
+
+
+*Figure 2. Turbopuffer’s object-storage queue batches contended CAS updates behind a broker. Source: [Turbopuffer object storage queue](https://turbopuffer.com/blog/object-storage-queue).*
+
+### SlateDB
+
+SlateDB is an embedded LSM-tree storage engine built for object storage. It stores durable state as object-store files: WAL SSTs, compacted SSTs, and manifest files. Like other LSM engines, it relies on immutable files for most data movement, but it still needs a safe coordination point to decide which writer owns the next part of the log.
+
+Conditional object creation is useful here because SlateDB can use object names as fencing points. Instead of overwriting a shared file, the writer creates the next manifest or WAL object with `If-None-Match: *`. The write succeeds only if that object does not already exist.
+
+
+
+*Figure 3. A SlateDB writer uses conditional object creation as a fencing point for object-store state.*
+
+### Iceberg file catalog
+
+Iceberg is another natural fit for this style of coordination. Iceberg table state is maintained in metadata files, and each commit creates a new metadata file. The commit succeeds by atomically swapping the table metadata pointer from the old metadata file to the new one. If another writer commits first, the swap fails, and the writer retries against the new table state.
+
+For an Iceberg-style file catalog on object storage, conditional requests can provide the missing compare-and-swap primitive. The catalog can store a small pointer object, such as `current.json`, that points to the latest table metadata file. A writer first writes the new metadata file using a unique name, then updates the current metadata pointer only if its ETag still matches the ETag that the writer read.
+
+
+
+*Figure 4. An Iceberg-style file catalog updates a small metadata pointer only if its ETag still matches.*
+
+### Leader election
+
+Leader election can also be built on top of conditional writes. The basic idea is to have all nodes compete to create the next lock file, such as `lock-0000000002.json`, using `If-None-Match: *`. Only one node can create the file successfully. That node becomes the leader for that epoch; the others receive a precondition failure and keep watching.
+
+
+
+*Figure 5. Nodes compete to create the next epoch lock file; only one writer succeeds.*
+
+Leader election by itself is not enough. A paused old leader can come back and still believe it owns the lock. This is the “zombie leader” problem. The fix is to use the leader epoch as a fencing token. Every request made by the leader includes the epoch, and downstream systems reject requests with an older epoch than the highest one they have already seen.
+
+
+
+*Figure 6. Epoch fencing prevents an old paused leader from writing after a newer leader takes over.*
+
+The leader should periodically update the lock file it acquired to signal liveness. Other nodes can poll the lock and check whether the lock was released or expired by looking at `Last-Modified`, which S3 exposes as standard object metadata.
+
+### WAL write/get with OSWALD
+
+OSWALD, the Object Storage Write-Ahead Log Device, shows how to build a WAL directly on object storage primitives. The design has three object types: a manifest object, snapshots, and log chunks. The manifest tracks the latest checkpoint and garbage collection progress; chunks hold the log content.
+
+Appending to the WAL can be done with conditional object creation. A writer creates the next chunk with `PUT If-None-Match`. If another writer already created that chunk for the same LSN, the write fails and the writer catches up by tailing the log. After creating a chunk, the writer also checks the manifest with `GET If-None-Match` to make sure garbage collection has not moved past its LSN before acknowledging the write.
+
+
+
+*Figure 7. OSWALD uses manifest, snapshot, and chunk objects to build a WAL on object storage. Source: [OSWALD](https://github.com/nvartolomei/oswald).*
+
+Now let’s look at how Ozone keeps S3 conditional requests—the CAS primitive—fast.
+
+## Technical details
+
+To keep the performance overhead minimal, thanks to [Ivan](https://github.com/ivandika3)’s suggestion, we coalesce the “conditional flag” with the normal client-to-server RPC message, so no additional RPC round trip is introduced. This characteristic is a great match for the optimistic path of conditional requests.
Review Comment:
**Grammar:** "Thanks to Ivan's suggestion" is dangling mid-sentence.
**Suggested:** "To keep performance overhead minimal—thanks to Ivan's suggestion—we coalesce the "conditional flag" with the normal client-to-server RPC message, so no additional RPC round trip is introduced."
Also: "This characteristic is a great match for the optimistic path" → "This fits well with the optimistic concurrency path."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
Review Comment:
**Grammar:** Missing preposition.
**Suggested:** "I also want to thank Xiangpeng for building the Parquet Viewer" or "Thanks also to Xiangpeng for building…"
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
+
+AWS also noted that conditional requests [let customers remove workaround code and simplify their systems](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=When%20we%20moved%20S3%20to,similar%20reaction). The same storage-level feature also [powers S3 Tables](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=they%20involve%20a%20complex,storage%2Dlevel%20features), which manages tabular data on S3.
+
+Common conditional request patterns include:
+
+- **Conditional `PUT`**
+ - Use `If-None-Match: *` to create an object only if it does not already exist.
+ - Use `If-Match: <etag>` to overwrite an object only if the current ETag is still the one the client observed.
+ - Use case: metadata commits, manifest updates, distributed lock files, and create-only log chunks.
+- **Conditional `GET`**
+ - Use `If-None-Match: <etag>` or date-based conditions to download the object only when it has changed.
+ - Use case: cache validation, dynamic model reloading, config refresh, and avoiding repeated downloads of large objects.
+- **Conditional `HEAD`**
+ - Same idea as conditional `GET`, but metadata-only.
+ - Use case: checking whether an object is still current without fetching the body.
+- **Conditional `POST` / `CompleteMultipartUpload`**
+ - Multipart upload writes data in parts, but the destination object becomes visible only when `CompleteMultipartUpload` succeeds.
+ - Conditional completion lets the client say: “complete this large object only if the destination still does not exist” or “only if the destination still matches this ETag.”
+ - Use case: large object writes where the final commit still needs create-only or compare-and-swap semantics.
+- **Conditional `COPY`**
+ - Source-side conditions make sure the source object is the version the client expects.
+ - Destination-side conditions make sure the copy does not overwrite unexpected data.
+ - Use case: snapshot promotion, clone/copy workflows, and metadata copy where both source and destination need validation.
+- **Conditional `DELETE`**
+ - Use `If-Match: <etag>` to delete only if the object is still unchanged.
+ - Use `If-Match: *` to delete only if the object exists.
+ - Use case: preventing a client from deleting an object that another writer has already replaced.
+
+For more advanced usage, let’s look at how modern systems use these APIs.
+
+## Application use cases
+
+So who exactly can use this?
+
+### Dynamic AI model reloading
+
+Advanced inference servers often cache massive model weights on local NVMe drives. To keep these models updated without constantly re-downloading them every few minutes, they can use S3 conditional requests. For example, the inference server can send a `GET` request with `If-None-Match` and the ETag of the model it already has.
+
+If the ETag still matches, Ozone can return `304 Not Modified`, so the server keeps using the local cached model and skips the download. If the ETag does not match, the model has been updated, so the server downloads the newest model and stores the new ETag for the next check. Thanks to [@amaliujia](https://github.com/amaliujia) for the example.
+
+
+
+*Figure 1. An inference server uses `If-None-Match` to avoid re-downloading unchanged model weights.*
+
+### Turbopuffer
+
+Turbopuffer’s object-storage queue is a great example of using a small metadata object to coordinate larger data work. In the simple version, a pusher reads `queue.json`, appends a job, and writes it back with CAS. A worker does the same thing to claim the next job. The CAS write only succeeds if `queue.json` has not changed since the client read it; otherwise the client reads the new queue and retries.
+
+This design is simple, but many clients can contend on one queue object. Turbopuffer improves it by batching updates with group commit, then moving the object-storage interaction behind a stateless broker. Clients talk to the broker, the broker performs one group-commit loop, and it only acknowledges work after the updated queue is durably committed.
+
+
+
+*Figure 2. Turbopuffer’s object-storage queue batches contended CAS updates behind a broker. Source: [Turbopuffer object storage queue](https://turbopuffer.com/blog/object-storage-queue).*
+
+### SlateDB
+
+SlateDB is an embedded LSM-tree storage engine built for object storage. It stores durable state as object-store files: WAL SSTs, compacted SSTs, and manifest files. Like other LSM engines, it relies on immutable files for most data movement, but it still needs a safe coordination point to decide which writer owns the next part of the log.
+
+Conditional object creation is useful here because SlateDB can use object names as fencing points. Instead of overwriting a shared file, the writer creates the next manifest or WAL object with `If-None-Match: *`. The write succeeds only if that object does not already exist.
+
+
+
+*Figure 3. A SlateDB writer uses conditional object creation as a fencing point for object-store state.*
+
+### Iceberg file catalog
+
+Iceberg is another natural fit for this style of coordination. Iceberg table state is maintained in metadata files, and each commit creates a new metadata file. The commit succeeds by atomically swapping the table metadata pointer from the old metadata file to the new one. If another writer commits first, the swap fails, and the writer retries against the new table state.
+
+For an Iceberg-style file catalog on object storage, conditional requests can provide the missing compare-and-swap primitive. The catalog can store a small pointer object, such as `current.json`, that points to the latest table metadata file. A writer first writes the new metadata file using a unique name, then updates the current metadata pointer only if its ETag still matches the ETag that the writer read.
+
+
+
+*Figure 4. An Iceberg-style file catalog updates a small metadata pointer only if its ETag still matches.*
+
+### Leader election
+
+Leader election can also be built on top of conditional writes. The basic idea is to have all nodes compete to create the next lock file, such as `lock-0000000002.json`, using `If-None-Match: *`. Only one node can create the file successfully. That node becomes the leader for that epoch; the others receive a precondition failure and keep watching.
+
+
+
+*Figure 5. Nodes compete to create the next epoch lock file; only one writer succeeds.*
+
+Leader election by itself is not enough. A paused old leader can come back and still believe it owns the lock. This is the “zombie leader” problem. The fix is to use the leader epoch as a fencing token. Every request made by the leader includes the epoch, and downstream systems reject requests with an older epoch than the highest one they have already seen.
+
+
+
+*Figure 6. Epoch fencing prevents an old paused leader from writing after a newer leader takes over.*
+
+The leader should periodically update the lock file it acquired to signal liveness. Other nodes can poll the lock and check whether the lock was released or expired by looking at `Last-Modified`, which S3 exposes as standard object metadata.
+
+### WAL write/get with OSWALD
+
+OSWALD, the Object Storage Write-Ahead Log Device, shows how to build a WAL directly on object storage primitives. The design has three object types: a manifest object, snapshots, and log chunks. The manifest tracks the latest checkpoint and garbage collection progress; chunks hold the log content.
+
+Appending to the WAL can be done with conditional object creation. A writer creates the next chunk with `PUT If-None-Match`. If another writer already created that chunk for the same LSN, the write fails and the writer catches up by tailing the log. After creating a chunk, the writer also checks the manifest with `GET If-None-Match` to make sure garbage collection has not moved past its LSN before acknowledging the write.
+
+
+
+*Figure 7. OSWALD uses manifest, snapshot, and chunk objects to build a WAL on object storage. Source: [OSWALD](https://github.com/nvartolomei/oswald).*
+
+Now let’s look at how Ozone keeps S3 conditional requests—the CAS primitive—fast.
+
+## Technical details
+
+To keep the performance overhead minimal, thanks to [Ivan](https://github.com/ivandika3)’s suggestion, we coalesce the “conditional flag” with the normal client-to-server RPC message, so no additional RPC round trip is introduced. This characteristic is a great match for the optimistic path of conditional requests.
+
+Before diving into that coalescing optimization, let’s look at Ozone’s atomic rewrite work, which is the foundation reused by S3 conditional requests.
+
+### Atomic rewrite
+
+
+
+*Figure 8. Atomic rewrite create and write phases.*
+
+The client indicates the expected epoch of the target object in the key-creation request. After Ozone Manager receives the `createKey` request, it first compares the expected epoch with the live key in the key table. If they match, OM stores the expected epoch along with the open key record in the open key table.
+
+At this stage, two clients can both successfully create an open key, so they can both start streaming file data to the datanode pipeline. Once a client believes the file data write is finished, it starts committing the key.
+
+:::note
+The client ID in `/vol1/bucket1/key1/{client_id}` is generated by Ozone Manager. It is a unique, non-human-readable ID that binds to a single key-creation lifecycle. It is not a persistent client ID. “Session ID” would probably be a better name, but the diagram uses a meaningful ID for clarity.
+:::
+
+During the commit phase, Ozone Manager compares the expected epoch stored in the client’s open key record with the live key epoch in the key table in the same transaction. If they match, OM overwrites the live key with the open key. If they do not match, OM cleans up the open key table and returns an atomic rewrite failure for the concurrent conflict.
+
+
+
+*Figure 9. Atomic rewrite commit phase.*
+
+### Coalescing the conditional flag
+
+In normal workloads, and especially under optimistic concurrency control, the successful path is expected to be common. Therefore, we validate the key’s ETag metadata at the same time as the key-creation request. Instead of introducing an additional request to fetch the latest key epoch and then plumbing that epoch into the atomic rewrite path, we extend atomic rewrite so it can recognize the expected ETag sent from S3 Gateway, validate it, translate it to an expected epoch, and store that epoch in the open key table. This lets the atomic rewrite commit path stay untouched.
+
+> For more information, see the discussion in [apache/ozone#9334](https://github.com/apache/ozone/pull/9334#discussion_r2558578333).
+
+
+
+*Figure 10. Coalescing the conditional flag into the normal Ozone write path.*
+
+### Implementation by API
+
+The discussion, design, and patches are available in Apache Jira and GitHub:
+
+- Epic ticket: [HDDS-13117](https://issues.apache.org/jira/browse/HDDS-13117)
+- Design: [`s3-conditional-requests.md`](https://github.com/apache/ozone/blob/master/hadoop-hdds/docs/content/design/s3-conditional-requests.md)
+
+#### PutObject
+
+Ozone Manager enforces conditional writes using a two-phase validation process, returning specific HTTP error codes depending on when a state mismatch is detected.
+
+During the initial `createKey` request, if OM determines that the expected ETag already mismatches the current state of the key, it immediately rejects the request with `412 Precondition Failed`.
+
+If the initial check passes but a concurrent modification occurs during the upload, OM detects the discrepancy during the `commitKey` phase. By comparing the expected epoch stored in the open session against the live key record in the key table, OM identifies the concurrent update and returns `409 Conflict`.
+
+Related patches:
+
+- [apache/ozone#10182](https://github.com/apache/ozone/pull/10182)
+- [apache/ozone#10023](https://github.com/apache/ozone/pull/10023)
+- [apache/ozone#9815](https://github.com/apache/ozone/pull/9815)
+
+##### `If-None-Match`
+
+We extend the existing atomic rewrite path to support the “generation must match `0`” condition, which matches GCS semantics. `If-None-Match: *` then reuses this create-if-absent path.
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: PUT object with If-None-Match = *
+ GW->>OM: createKey(expectedDataGeneration = 0)
+ OM->>OM: Reject if key already exists
+ OM-->>GW: Open key or KEY_ALREADY_EXISTS
+ opt Open key created
+ GW->>OM: commitKey()
+ OM->>OM: Recheck key absence during commit
+ OM-->>GW: Success or generation mismatch
+ end
+ GW-->>User: 200 OK or 412 Precondition Failed
+```
+
+*Figure 11. `PUT If-None-Match` maps create-if-absent semantics to Ozone’s generation-match path.*
+
+##### `If-Match`
+
+S3 Gateway carries the expected ETag along with the `createKey` request. OM compares it with the live key, translates it to the expected epoch recognized by atomic rewrite, and then reuses the existing commit path.
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: PUT object with If-Match ETag
+ Note over GW,OM: No pre-read for current ETag/updateID on optimistic path
+ GW->>OM: createKey(expectedETag = ETag)
+ OM->>OM: Validate expected ETag against live key
+ alt ETag already unmatched
+ OM-->>GW: ETAG_MISMATCH or KEY_NOT_FOUND
+ GW-->>User: 412 Precondition Failed
+ else ETag matches
+ OM->>OM: Derive updateID and persist expected epoch
+ OM-->>GW: Open key with resolved expected epoch
+ GW->>OM: commitKey()
+ OM->>OM: Compare open session epoch with live key record
+ alt Epoch mismatch
+ OM-->>GW: GENERATION_MISMATCH
+ GW-->>User: 409 ConditionalRequestConflict
+ else Epoch matches
+ OM->>OM: Commit successful
+ OM-->>GW: Success
+ GW-->>User: 200 OK
+ end
+ end
+```
+
+*Figure 12. `PUT If-Match` validates the ETag once, stores the matched epoch, and rechecks that epoch at commit time.*
+
+#### GetObject / HeadObject
+
+For `GetObject` and `HeadObject`, S3 Gateway fetches key metadata from OM and evaluates the conditional headers against that metadata.
+
+Related patch: [apache/ozone#10031](https://github.com/apache/ozone/pull/10031)
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: GET/HEAD with conditional headers
+ GW->>OM: getS3KeyDetails() or headS3Object()
+ OM-->>GW: ETag, modificationTime, and key info
+ GW->>GW: Evaluate ETag/date precedence rules
+ alt Not modified
+ GW-->>User: 304 Not Modified
+ else Precondition failed
+ GW-->>User: 412 Precondition Failed
+ else Preconditions pass
+ opt GET or ranged GET
+ GW->>OM: getKey() or open data stream
+ OM-->>GW: Object data
+ end
+ GW-->>User: 200 OK or 206 Partial Content
+ end
+```
+
+*Figure 13. `GET` and `HEAD` evaluate conditional headers after retrieving object metadata from OM.*
+
+#### CopyObject
+
+`CopyObject` is effectively `GetObject` plus `PutObject`, because it must enforce conditional logic on both the source and destination. During the source phase, the gateway resolves the source object metadata and evaluates preconditions against that snapshot before streaming the data. In the destination phase, the gateway performs a conditional write to the destination, reusing the atomic write APIs introduced for `PutObject`.
+
+Related patch: [apache/ozone#10207](https://github.com/apache/ozone/pull/10207). Thanks [@YutaLin](https://github.com/YutaLin).
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: COPY with source and destination conditions
+ GW->>OM: Get source key details
+ OM-->>GW: Source metadata and bound content stream
+ GW->>GW: Evaluate x-amz-copy-source-if-* headers
+ alt Source precondition failed
+ GW-->>User: 412 Precondition Failed
+ else Source passes
+ GW->>OM: Create destination key with optional conditions
+ OM->>OM: Validate destination state and open key
+ OM-->>GW: Open key or 412-mapped error
+ opt Destination open key created
+ GW->>OM: Commit destination key using source snapshot
+ OM->>OM: Revalidate generation for conditional writes
+ OM-->>GW: Success or 412-mapped error
+ end
+ GW-->>User: 200 OK or 412 Precondition Failed
+ end
+```
+
+*Figure 14. `CopyObject` checks source preconditions first, then reuses conditional destination writes.*
+
+#### CompleteMultipartUpload
+
+`CompleteMultipartUpload` compares the caller’s expected ETag or create-if-absent condition with the current database record on the server side.
+
+Related patch: [apache/ozone#10164](https://github.com/apache/ozone/pull/10164). Thanks [@YutaLin](https://github.com/YutaLin).
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: CompleteMultipartUpload with optional conditions
+ GW->>GW: Parse conditional write headers
+ GW->>OM: completeMultipartUpload(keyArgs + partsList)
+ OM->>OM: Acquire bucket lock
+ OM->>OM: Load current committed key and MPU state
+ OM->>OM: Validate destination precondition
+ alt Precondition failed
+ OM-->>GW: KEY_ALREADY_EXISTS, KEY_NOT_FOUND, or ETAG_* error
+ GW-->>User: 412 Precondition Failed
+ else Preconditions pass
+ OM->>OM: Validate MPU parts and build final key
+ OM->>OM: Write key table entry and remove MPU state
+ OM-->>GW: Success
+ GW-->>User: 200 OK
+ end
+```
+
+*Figure 15. Conditional `CompleteMultipartUpload` validates the destination state before publishing the completed object.*
+
+#### DeleteObject
+
+`DeleteObject` compares the caller’s expected ETag with the current database record on the server side before writing the delete tombstone.
+
+Related patch: [apache/ozone#10511](https://github.com/apache/ozone/pull/10511)
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: DELETE with optional If-Match
+ alt No If-Match
+ GW->>OM: deleteKey()
+ OM->>OM: Apply existing delete semantics
+ OM-->>GW: Success or KEY_NOT_FOUND
+ GW-->>User: 204 No Content
+ else If-Match present
+ GW->>OM: deleteKey(expectedETag or *)
+ OM->>OM: Validate current key under lock and write tombstone
+ OM-->>GW: Success, KEY_NOT_FOUND, or ETAG_* error
+ GW-->>User: 204 No Content or 412 Precondition Failed
+ end
+```
+
+*Figure 16. Conditional `DeleteObject` deletes only when the current object still matches the expected ETag.*
+
+## Benchmark
+
+The main performance question for conditional writes is whether adding the precondition makes the common successful path slower than a normal write. In Ozone, the expected ETag or create-if-absent flag is carried with the normal S3 Gateway to OM request, then validated in OM’s metadata update path. This design avoids a separate gateway-side metadata read before every conditional write.
+
+To sanity-check that assumption, we ran a small local benchmark against a Docker Compose Ozone cluster through S3 Gateway. The benchmark used 4 KiB objects, 15 warmup iterations, and 100 measured iterations per operation. It compared four operations:
+
+- normal `PUT` creating a new key,
+- `PUT If-None-Match: *` creating a new key,
+- normal `PUT` overwriting an existing key,
+- `PUT If-Match: <etag>` overwriting an existing key.
+
+
+
+*Figure 17. Local benchmark latency for normal `PUT`, `PUT If-None-Match`, normal overwrite, and `PUT If-Match`.*
+
+In this setup, conditional create was effectively the same as normal create: the mean latency differed by about 0.12 ms, and the p95 was slightly lower. Conditional overwrite was also close to normal overwrite: the mean latency was about 0.35 ms higher, while p95 remained slightly lower and p99 was nearly identical.
+
+## Conclusion
+
+Object storage has quietly crossed a line. It is no longer just where systems park their bytes. It has become the place where they coordinate. Metadata catalogs, write-ahead logs, leader election, job queues, model caches: all of these need a single point of truth about “who wrote last,” and increasingly they want to ask that question of the storage layer itself.
+
+By bringing native S3 conditional requests to Apache Ozone, and doing it without a single extra RPC on the happy path. We turn Ozone into a first-class substrate for optimistic concurrency control. Applications can compare-and-swap directly against the objects they already store, and retire the external lock services, catalog databases, and consensus clusters they once needed just to avoid a lost update.
Review Comment:
**Grammar:** This is a sentence fragment — the participial phrase ends with a period, then a new sentence starts with "We turn…"
**Suggested:** "By bringing native S3 conditional requests to Apache Ozone without adding an extra RPC on the happy path, we turn Ozone into a first-class substrate for optimistic concurrency control."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
Review Comment:
**Suggested:** "at the time of writing" or "when this article was written" (drop "that").
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
+
+1. First, we originally used one JSON file to represent one run result. Each latest run was about 0.236 MiB. This sounds small, but the problem is that we did not lazy load these files at the beginning. Every time the page refreshed, even if some files might already be cached by the browser, the TypeScript data structure still ended up holding too much content from too many files. That is very memory-consuming.
Review Comment:
**Chinglish:** Two spots on this line.
- "at the beginning" → "initially" / "on first load"
- "memory-consuming" → "used too much memory" / "memory-intensive"
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
Review Comment:
**Suggested:** `## How to Use the Dashboard` — question marks in headings are unusual in English blog posts.
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
+
+1. First, we originally used one JSON file to represent one run result. Each latest run was about 0.236 MiB. This sounds small, but the problem is that we did not lazy load these files at the beginning. Every time the page refreshed, even if some files might already be cached by the browser, the TypeScript data structure still ended up holding too much content from too many files. That is very memory-consuming.
+
+2. Second, the JSON data had a ton of room to compress. The data shape is very repetitive: `run_id`, `suite_key`, `status`, feature names, source paths, class names, test names, and similar object keys appear again and again. JSON stores all of that as repeated text and nested objects. Even if HTTP compression helps over the network, the browser still has to parse it into JavaScript objects, and Git still stores changed blobs. This kind of data is much closer to a table than a document, so a columnar format makes more sense.
+
+3. Finally, the search index data was not normalized correctly. The index repeated run metadata and case metadata too much, and it grew quickly. For Git history, this is especially bad because every new run changes a large generated file again.
+
+So then I basically applied three optimizations: lazy loading, Parquet compression, and search/index normalization.
+
+1. **Lazy loading.** The page now loads the light catalog/index first. Full run details are fetched only when you open the latest run or expand an archived run. Search index loading is also delayed until you actually search, and the browser can keep the FlexSearch index in IndexedDB. Logs are fetched only when you open a log modal. Source snippets are fetched only when you open a case. This makes the first page load much cheaper.
+2. **Compress data as Parquet.** Instead of treating every run as one large JSON document, the report data is split into Parquet tables: catalog runs, suites, features, per-run metadata, per-run cases, search rows, log files, and log lines. The Parquet writer uses Zstd compression and dictionary encoding, which is a much better fit for repeated values like suite names, statuses, feature tags, and source paths. The frontend then queries these Parquet files in the browser through DuckDB-Wasm.
+3. **Index file optimization.** The search rows now contain normalized fields and a short `detail_preview`, while the full case detail can be hydrated from the per-run case Parquet file when needed. For JSON fallback, the index and search data are partitioned into smaller shards. For the normal Parquet path, the global search index is a compressed Parquet file. The search index also has a stable `index_id`, so the browser can reuse the IndexedDB cache when the data has not changed.
+
+Result is crazy:
+
+1. Parquet data now:
+ 1. search rows: 7.64 MiB, 33,876 rows, growing about 0.142-0.147 MiB/run
+ 2. run case/detail data excluding search: 4.92 MiB across 52 runs, growing about 0.093 MiB/run
+ 3. catalog/index data: 0.03 MiB now, growing less than 1 KiB/run
+ 4. total Pages data: 12.59 MiB across 52 runs
+ 5. latest run data: 246,696 bytes \= 0.235 MiB
+2. Projection at the current rate:
+ 1. about \+0.236 MiB/run
+ 2. about \+7.1 MiB/month
+ 3. about \+86 MiB/year
+2. Compared with the JSON baseline:
+ 1. current data size: 149.11 MiB \-\> 12.59 MiB, about 11.8x smaller
+ 2. yearly growth: \~1.0 GiB/year \-\> \~86 MiB/year, about 12x slower
+ 3. search data: 107.09 MiB \-\> 7.64 MiB, about 14x smaller
+
+## Current Effect
Review Comment:
**Chinglish:** "Current Effect" ≈ 当前效果.
**Suggested:** `## Impact` or `## Results So Far`.
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
+
+AWS also noted that conditional requests [let customers remove workaround code and simplify their systems](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=When%20we%20moved%20S3%20to,similar%20reaction). The same storage-level feature also [powers S3 Tables](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=they%20involve%20a%20complex,storage%2Dlevel%20features), which manages tabular data on S3.
+
+Common conditional request patterns include:
+
+- **Conditional `PUT`**
+ - Use `If-None-Match: *` to create an object only if it does not already exist.
+ - Use `If-Match: <etag>` to overwrite an object only if the current ETag is still the one the client observed.
+ - Use case: metadata commits, manifest updates, distributed lock files, and create-only log chunks.
+- **Conditional `GET`**
+ - Use `If-None-Match: <etag>` or date-based conditions to download the object only when it has changed.
+ - Use case: cache validation, dynamic model reloading, config refresh, and avoiding repeated downloads of large objects.
+- **Conditional `HEAD`**
+ - Same idea as conditional `GET`, but metadata-only.
+ - Use case: checking whether an object is still current without fetching the body.
+- **Conditional `POST` / `CompleteMultipartUpload`**
+ - Multipart upload writes data in parts, but the destination object becomes visible only when `CompleteMultipartUpload` succeeds.
+ - Conditional completion lets the client say: “complete this large object only if the destination still does not exist” or “only if the destination still matches this ETag.”
+ - Use case: large object writes where the final commit still needs create-only or compare-and-swap semantics.
+- **Conditional `COPY`**
+ - Source-side conditions make sure the source object is the version the client expects.
+ - Destination-side conditions make sure the copy does not overwrite unexpected data.
+ - Use case: snapshot promotion, clone/copy workflows, and metadata copy where both source and destination need validation.
+- **Conditional `DELETE`**
+ - Use `If-Match: <etag>` to delete only if the object is still unchanged.
+ - Use `If-Match: *` to delete only if the object exists.
+ - Use case: preventing a client from deleting an object that another writer has already replaced.
+
+For more advanced usage, let’s look at how modern systems use these APIs.
+
+## Application use cases
+
+So who exactly can use this?
+
+### Dynamic AI model reloading
+
+Advanced inference servers often cache massive model weights on local NVMe drives. To keep these models updated without constantly re-downloading them every few minutes, they can use S3 conditional requests. For example, the inference server can send a `GET` request with `If-None-Match` and the ETag of the model it already has.
+
+If the ETag still matches, Ozone can return `304 Not Modified`, so the server keeps using the local cached model and skips the download. If the ETag does not match, the model has been updated, so the server downloads the newest model and stores the new ETag for the next check. Thanks to [@amaliujia](https://github.com/amaliujia) for the example.
+
+
+
+*Figure 1. An inference server uses `If-None-Match` to avoid re-downloading unchanged model weights.*
+
+### Turbopuffer
+
+Turbopuffer’s object-storage queue is a great example of using a small metadata object to coordinate larger data work. In the simple version, a pusher reads `queue.json`, appends a job, and writes it back with CAS. A worker does the same thing to claim the next job. The CAS write only succeeds if `queue.json` has not changed since the client read it; otherwise the client reads the new queue and retries.
+
+This design is simple, but many clients can contend on one queue object. Turbopuffer improves it by batching updates with group commit, then moving the object-storage interaction behind a stateless broker. Clients talk to the broker, the broker performs one group-commit loop, and it only acknowledges work after the updated queue is durably committed.
+
+
+
+*Figure 2. Turbopuffer’s object-storage queue batches contended CAS updates behind a broker. Source: [Turbopuffer object storage queue](https://turbopuffer.com/blog/object-storage-queue).*
+
+### SlateDB
+
+SlateDB is an embedded LSM-tree storage engine built for object storage. It stores durable state as object-store files: WAL SSTs, compacted SSTs, and manifest files. Like other LSM engines, it relies on immutable files for most data movement, but it still needs a safe coordination point to decide which writer owns the next part of the log.
+
+Conditional object creation is useful here because SlateDB can use object names as fencing points. Instead of overwriting a shared file, the writer creates the next manifest or WAL object with `If-None-Match: *`. The write succeeds only if that object does not already exist.
+
+
+
+*Figure 3. A SlateDB writer uses conditional object creation as a fencing point for object-store state.*
+
+### Iceberg file catalog
+
+Iceberg is another natural fit for this style of coordination. Iceberg table state is maintained in metadata files, and each commit creates a new metadata file. The commit succeeds by atomically swapping the table metadata pointer from the old metadata file to the new one. If another writer commits first, the swap fails, and the writer retries against the new table state.
+
+For an Iceberg-style file catalog on object storage, conditional requests can provide the missing compare-and-swap primitive. The catalog can store a small pointer object, such as `current.json`, that points to the latest table metadata file. A writer first writes the new metadata file using a unique name, then updates the current metadata pointer only if its ETag still matches the ETag that the writer read.
+
+
+
+*Figure 4. An Iceberg-style file catalog updates a small metadata pointer only if its ETag still matches.*
+
+### Leader election
+
+Leader election can also be built on top of conditional writes. The basic idea is to have all nodes compete to create the next lock file, such as `lock-0000000002.json`, using `If-None-Match: *`. Only one node can create the file successfully. That node becomes the leader for that epoch; the others receive a precondition failure and keep watching.
+
+
+
+*Figure 5. Nodes compete to create the next epoch lock file; only one writer succeeds.*
+
+Leader election by itself is not enough. A paused old leader can come back and still believe it owns the lock. This is the “zombie leader” problem. The fix is to use the leader epoch as a fencing token. Every request made by the leader includes the epoch, and downstream systems reject requests with an older epoch than the highest one they have already seen.
+
+
+
+*Figure 6. Epoch fencing prevents an old paused leader from writing after a newer leader takes over.*
+
+The leader should periodically update the lock file it acquired to signal liveness. Other nodes can poll the lock and check whether the lock was released or expired by looking at `Last-Modified`, which S3 exposes as standard object metadata.
+
+### WAL write/get with OSWALD
Review Comment:
**Suggested:** `### WAL Writes and Reads with OSWALD` (instead of "write/get").
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
+
+AWS also noted that conditional requests [let customers remove workaround code and simplify their systems](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=When%20we%20moved%20S3%20to,similar%20reaction). The same storage-level feature also [powers S3 Tables](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=they%20involve%20a%20complex,storage%2Dlevel%20features), which manages tabular data on S3.
+
+Common conditional request patterns include:
+
+- **Conditional `PUT`**
+ - Use `If-None-Match: *` to create an object only if it does not already exist.
+ - Use `If-Match: <etag>` to overwrite an object only if the current ETag is still the one the client observed.
+ - Use case: metadata commits, manifest updates, distributed lock files, and create-only log chunks.
+- **Conditional `GET`**
+ - Use `If-None-Match: <etag>` or date-based conditions to download the object only when it has changed.
+ - Use case: cache validation, dynamic model reloading, config refresh, and avoiding repeated downloads of large objects.
+- **Conditional `HEAD`**
+ - Same idea as conditional `GET`, but metadata-only.
+ - Use case: checking whether an object is still current without fetching the body.
+- **Conditional `POST` / `CompleteMultipartUpload`**
+ - Multipart upload writes data in parts, but the destination object becomes visible only when `CompleteMultipartUpload` succeeds.
+ - Conditional completion lets the client say: “complete this large object only if the destination still does not exist” or “only if the destination still matches this ETag.”
+ - Use case: large object writes where the final commit still needs create-only or compare-and-swap semantics.
+- **Conditional `COPY`**
+ - Source-side conditions make sure the source object is the version the client expects.
+ - Destination-side conditions make sure the copy does not overwrite unexpected data.
+ - Use case: snapshot promotion, clone/copy workflows, and metadata copy where both source and destination need validation.
+- **Conditional `DELETE`**
+ - Use `If-Match: <etag>` to delete only if the object is still unchanged.
+ - Use `If-Match: *` to delete only if the object exists.
+ - Use case: preventing a client from deleting an object that another writer has already replaced.
+
+For more advanced usage, let’s look at how modern systems use these APIs.
+
+## Application use cases
+
+So who exactly can use this?
+
+### Dynamic AI model reloading
+
+Advanced inference servers often cache massive model weights on local NVMe drives. To keep these models updated without constantly re-downloading them every few minutes, they can use S3 conditional requests. For example, the inference server can send a `GET` request with `If-None-Match` and the ETag of the model it already has.
+
+If the ETag still matches, Ozone can return `304 Not Modified`, so the server keeps using the local cached model and skips the download. If the ETag does not match, the model has been updated, so the server downloads the newest model and stores the new ETag for the next check. Thanks to [@amaliujia](https://github.com/amaliujia) for the example.
+
+
+
+*Figure 1. An inference server uses `If-None-Match` to avoid re-downloading unchanged model weights.*
+
+### Turbopuffer
+
+Turbopuffer’s object-storage queue is a great example of using a small metadata object to coordinate larger data work. In the simple version, a pusher reads `queue.json`, appends a job, and writes it back with CAS. A worker does the same thing to claim the next job. The CAS write only succeeds if `queue.json` has not changed since the client read it; otherwise the client reads the new queue and retries.
+
+This design is simple, but many clients can contend on one queue object. Turbopuffer improves it by batching updates with group commit, then moving the object-storage interaction behind a stateless broker. Clients talk to the broker, the broker performs one group-commit loop, and it only acknowledges work after the updated queue is durably committed.
+
+
+
+*Figure 2. Turbopuffer’s object-storage queue batches contended CAS updates behind a broker. Source: [Turbopuffer object storage queue](https://turbopuffer.com/blog/object-storage-queue).*
+
+### SlateDB
+
+SlateDB is an embedded LSM-tree storage engine built for object storage. It stores durable state as object-store files: WAL SSTs, compacted SSTs, and manifest files. Like other LSM engines, it relies on immutable files for most data movement, but it still needs a safe coordination point to decide which writer owns the next part of the log.
+
+Conditional object creation is useful here because SlateDB can use object names as fencing points. Instead of overwriting a shared file, the writer creates the next manifest or WAL object with `If-None-Match: *`. The write succeeds only if that object does not already exist.
+
+
+
+*Figure 3. A SlateDB writer uses conditional object creation as a fencing point for object-store state.*
+
+### Iceberg file catalog
+
+Iceberg is another natural fit for this style of coordination. Iceberg table state is maintained in metadata files, and each commit creates a new metadata file. The commit succeeds by atomically swapping the table metadata pointer from the old metadata file to the new one. If another writer commits first, the swap fails, and the writer retries against the new table state.
+
+For an Iceberg-style file catalog on object storage, conditional requests can provide the missing compare-and-swap primitive. The catalog can store a small pointer object, such as `current.json`, that points to the latest table metadata file. A writer first writes the new metadata file using a unique name, then updates the current metadata pointer only if its ETag still matches the ETag that the writer read.
+
+
+
+*Figure 4. An Iceberg-style file catalog updates a small metadata pointer only if its ETag still matches.*
+
+### Leader election
+
+Leader election can also be built on top of conditional writes. The basic idea is to have all nodes compete to create the next lock file, such as `lock-0000000002.json`, using `If-None-Match: *`. Only one node can create the file successfully. That node becomes the leader for that epoch; the others receive a precondition failure and keep watching.
+
+
+
+*Figure 5. Nodes compete to create the next epoch lock file; only one writer succeeds.*
+
+Leader election by itself is not enough. A paused old leader can come back and still believe it owns the lock. This is the “zombie leader” problem. The fix is to use the leader epoch as a fencing token. Every request made by the leader includes the epoch, and downstream systems reject requests with an older epoch than the highest one they have already seen.
+
+
+
+*Figure 6. Epoch fencing prevents an old paused leader from writing after a newer leader takes over.*
+
+The leader should periodically update the lock file it acquired to signal liveness. Other nodes can poll the lock and check whether the lock was released or expired by looking at `Last-Modified`, which S3 exposes as standard object metadata.
+
+### WAL write/get with OSWALD
+
+OSWALD, the Object Storage Write-Ahead Log Device, shows how to build a WAL directly on object storage primitives. The design has three object types: a manifest object, snapshots, and log chunks. The manifest tracks the latest checkpoint and garbage collection progress; chunks hold the log content.
+
+Appending to the WAL can be done with conditional object creation. A writer creates the next chunk with `PUT If-None-Match`. If another writer already created that chunk for the same LSN, the write fails and the writer catches up by tailing the log. After creating a chunk, the writer also checks the manifest with `GET If-None-Match` to make sure garbage collection has not moved past its LSN before acknowledging the write.
+
+
+
+*Figure 7. OSWALD uses manifest, snapshot, and chunk objects to build a WAL on object storage. Source: [OSWALD](https://github.com/nvartolomei/oswald).*
+
+Now let’s look at how Ozone keeps S3 conditional requests—the CAS primitive—fast.
+
+## Technical details
+
+To keep the performance overhead minimal, thanks to [Ivan](https://github.com/ivandika3)’s suggestion, we coalesce the “conditional flag” with the normal client-to-server RPC message, so no additional RPC round trip is introduced. This characteristic is a great match for the optimistic path of conditional requests.
+
+Before diving into that coalescing optimization, let’s look at Ozone’s atomic rewrite work, which is the foundation reused by S3 conditional requests.
+
+### Atomic rewrite
+
+
+
+*Figure 8. Atomic rewrite create and write phases.*
+
+The client indicates the expected epoch of the target object in the key-creation request. After Ozone Manager receives the `createKey` request, it first compares the expected epoch with the live key in the key table. If they match, OM stores the expected epoch along with the open key record in the open key table.
+
+At this stage, two clients can both successfully create an open key, so they can both start streaming file data to the datanode pipeline. Once a client believes the file data write is finished, it starts committing the key.
+
+:::note
+The client ID in `/vol1/bucket1/key1/{client_id}` is generated by Ozone Manager. It is a unique, non-human-readable ID that binds to a single key-creation lifecycle. It is not a persistent client ID. “Session ID” would probably be a better name, but the diagram uses a meaningful ID for clarity.
+:::
+
+During the commit phase, Ozone Manager compares the expected epoch stored in the client’s open key record with the live key epoch in the key table in the same transaction. If they match, OM overwrites the live key with the open key. If they do not match, OM cleans up the open key table and returns an atomic rewrite failure for the concurrent conflict.
+
+
+
+*Figure 9. Atomic rewrite commit phase.*
+
+### Coalescing the conditional flag
+
+In normal workloads, and especially under optimistic concurrency control, the successful path is expected to be common. Therefore, we validate the key’s ETag metadata at the same time as the key-creation request. Instead of introducing an additional request to fetch the latest key epoch and then plumbing that epoch into the atomic rewrite path, we extend atomic rewrite so it can recognize the expected ETag sent from S3 Gateway, validate it, translate it to an expected epoch, and store that epoch in the open key table. This lets the atomic rewrite commit path stay untouched.
+
+> For more information, see the discussion in [apache/ozone#9334](https://github.com/apache/ozone/pull/9334#discussion_r2558578333).
+
+
+
+*Figure 10. Coalescing the conditional flag into the normal Ozone write path.*
+
+### Implementation by API
+
+The discussion, design, and patches are available in Apache Jira and GitHub:
+
+- Epic ticket: [HDDS-13117](https://issues.apache.org/jira/browse/HDDS-13117)
+- Design: [`s3-conditional-requests.md`](https://github.com/apache/ozone/blob/master/hadoop-hdds/docs/content/design/s3-conditional-requests.md)
+
+#### PutObject
+
+Ozone Manager enforces conditional writes using a two-phase validation process, returning specific HTTP error codes depending on when a state mismatch is detected.
+
+During the initial `createKey` request, if OM determines that the expected ETag already mismatches the current state of the key, it immediately rejects the request with `412 Precondition Failed`.
+
+If the initial check passes but a concurrent modification occurs during the upload, OM detects the discrepancy during the `commitKey` phase. By comparing the expected epoch stored in the open session against the live key record in the key table, OM identifies the concurrent update and returns `409 Conflict`.
+
+Related patches:
+
+- [apache/ozone#10182](https://github.com/apache/ozone/pull/10182)
+- [apache/ozone#10023](https://github.com/apache/ozone/pull/10023)
+- [apache/ozone#9815](https://github.com/apache/ozone/pull/9815)
+
+##### `If-None-Match`
+
+We extend the existing atomic rewrite path to support the “generation must match `0`” condition, which matches GCS semantics. `If-None-Match: *` then reuses this create-if-absent path.
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: PUT object with If-None-Match = *
+ GW->>OM: createKey(expectedDataGeneration = 0)
+ OM->>OM: Reject if key already exists
+ OM-->>GW: Open key or KEY_ALREADY_EXISTS
+ opt Open key created
+ GW->>OM: commitKey()
+ OM->>OM: Recheck key absence during commit
+ OM-->>GW: Success or generation mismatch
+ end
+ GW-->>User: 200 OK or 412 Precondition Failed
+```
+
+*Figure 11. `PUT If-None-Match` maps create-if-absent semantics to Ozone’s generation-match path.*
+
+##### `If-Match`
+
+S3 Gateway carries the expected ETag along with the `createKey` request. OM compares it with the live key, translates it to the expected epoch recognized by atomic rewrite, and then reuses the existing commit path.
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: PUT object with If-Match ETag
+ Note over GW,OM: No pre-read for current ETag/updateID on optimistic path
+ GW->>OM: createKey(expectedETag = ETag)
+ OM->>OM: Validate expected ETag against live key
+ alt ETag already unmatched
+ OM-->>GW: ETAG_MISMATCH or KEY_NOT_FOUND
+ GW-->>User: 412 Precondition Failed
+ else ETag matches
+ OM->>OM: Derive updateID and persist expected epoch
+ OM-->>GW: Open key with resolved expected epoch
+ GW->>OM: commitKey()
+ OM->>OM: Compare open session epoch with live key record
+ alt Epoch mismatch
+ OM-->>GW: GENERATION_MISMATCH
+ GW-->>User: 409 ConditionalRequestConflict
+ else Epoch matches
+ OM->>OM: Commit successful
+ OM-->>GW: Success
+ GW-->>User: 200 OK
+ end
+ end
+```
+
+*Figure 12. `PUT If-Match` validates the ETag once, stores the matched epoch, and rechecks that epoch at commit time.*
+
+#### GetObject / HeadObject
+
+For `GetObject` and `HeadObject`, S3 Gateway fetches key metadata from OM and evaluates the conditional headers against that metadata.
+
+Related patch: [apache/ozone#10031](https://github.com/apache/ozone/pull/10031)
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: GET/HEAD with conditional headers
+ GW->>OM: getS3KeyDetails() or headS3Object()
+ OM-->>GW: ETag, modificationTime, and key info
+ GW->>GW: Evaluate ETag/date precedence rules
+ alt Not modified
+ GW-->>User: 304 Not Modified
+ else Precondition failed
+ GW-->>User: 412 Precondition Failed
+ else Preconditions pass
+ opt GET or ranged GET
+ GW->>OM: getKey() or open data stream
+ OM-->>GW: Object data
+ end
+ GW-->>User: 200 OK or 206 Partial Content
+ end
+```
+
+*Figure 13. `GET` and `HEAD` evaluate conditional headers after retrieving object metadata from OM.*
+
+#### CopyObject
+
+`CopyObject` is effectively `GetObject` plus `PutObject`, because it must enforce conditional logic on both the source and destination. During the source phase, the gateway resolves the source object metadata and evaluates preconditions against that snapshot before streaming the data. In the destination phase, the gateway performs a conditional write to the destination, reusing the atomic write APIs introduced for `PutObject`.
+
+Related patch: [apache/ozone#10207](https://github.com/apache/ozone/pull/10207). Thanks [@YutaLin](https://github.com/YutaLin).
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant GW as S3 Gateway
+ participant OM as Ozone Manager
+
+ User->>GW: COPY with source and destination conditions
+ GW->>OM: Get source key details
+ OM-->>GW: Source metadata and bound content stream
+ GW->>GW: Evaluate x-amz-copy-source-if-* headers
+ alt Source precondition failed
+ GW-->>User: 412 Precondition Failed
+ else Source passes
+ GW->>OM: Create destination key with optional conditions
+ OM->>OM: Validate destination state and open key
+ OM-->>GW: Open key or 412-mapped error
+ opt Destination open key created
+ GW->>OM: Commit destination key using source snapshot
+ OM->>OM: Revalidate generation for conditional writes
+ OM-->>GW: Success or 412-mapped error
+ end
+ GW-->>User: 200 OK or 412 Precondition Failed
+ end
+```
+
+*Figure 14. `CopyObject` checks source preconditions first, then reuses conditional destination writes.*
+
+#### CompleteMultipartUpload
+
+`CompleteMultipartUpload` compares the caller’s expected ETag or create-if-absent condition with the current database record on the server side.
Review Comment:
**Suggested:** "current metadata record" or "committed key state" (instead of "database record on the server side"). Same note applies to the `DeleteObject` section (~line 354).
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
Review Comment:
**Suggested:** "Apache Ozone had already invested considerable effort in tracking" (instead of "tried really hard").
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
Review Comment:
**Chinglish:** "at the beginning" ≈ 一开始.
**Suggested:** "does not load all case details upfront" or "on initial load."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
Review Comment:
**Chinglish:** "really" here reads like 真的. Also duplicated in Insight #2 — consider keeping only one PMC-promotion mention.
**Suggested:** "This helped me earn PMC membership" or "This played a role in my promotion to PMC."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
Review Comment:
**Suggested:** "But if you click through those links" — drop "I believe" and "really."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
+
+1. First, we originally used one JSON file to represent one run result. Each latest run was about 0.236 MiB. This sounds small, but the problem is that we did not lazy load these files at the beginning. Every time the page refreshed, even if some files might already be cached by the browser, the TypeScript data structure still ended up holding too much content from too many files. That is very memory-consuming.
+
+2. Second, the JSON data had a ton of room to compress. The data shape is very repetitive: `run_id`, `suite_key`, `status`, feature names, source paths, class names, test names, and similar object keys appear again and again. JSON stores all of that as repeated text and nested objects. Even if HTTP compression helps over the network, the browser still has to parse it into JavaScript objects, and Git still stores changed blobs. This kind of data is much closer to a table than a document, so a columnar format makes more sense.
+
+3. Finally, the search index data was not normalized correctly. The index repeated run metadata and case metadata too much, and it grew quickly. For Git history, this is especially bad because every new run changes a large generated file again.
+
+So then I basically applied three optimizations: lazy loading, Parquet compression, and search/index normalization.
+
+1. **Lazy loading.** The page now loads the light catalog/index first. Full run details are fetched only when you open the latest run or expand an archived run. Search index loading is also delayed until you actually search, and the browser can keep the FlexSearch index in IndexedDB. Logs are fetched only when you open a log modal. Source snippets are fetched only when you open a case. This makes the first page load much cheaper.
+2. **Compress data as Parquet.** Instead of treating every run as one large JSON document, the report data is split into Parquet tables: catalog runs, suites, features, per-run metadata, per-run cases, search rows, log files, and log lines. The Parquet writer uses Zstd compression and dictionary encoding, which is a much better fit for repeated values like suite names, statuses, feature tags, and source paths. The frontend then queries these Parquet files in the browser through DuckDB-Wasm.
+3. **Index file optimization.** The search rows now contain normalized fields and a short `detail_preview`, while the full case detail can be hydrated from the per-run case Parquet file when needed. For JSON fallback, the index and search data are partitioned into smaller shards. For the normal Parquet path, the global search index is a compressed Parquet file. The search index also has a stable `index_id`, so the browser can reuse the IndexedDB cache when the data has not changed.
+
+Result is crazy:
Review Comment:
**Suggested:** "The results were dramatic:" or "The improvements were significant:" (instead of "Result is crazy:").
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
+
+1. First, we originally used one JSON file to represent one run result. Each latest run was about 0.236 MiB. This sounds small, but the problem is that we did not lazy load these files at the beginning. Every time the page refreshed, even if some files might already be cached by the browser, the TypeScript data structure still ended up holding too much content from too many files. That is very memory-consuming.
+
+2. Second, the JSON data had a ton of room to compress. The data shape is very repetitive: `run_id`, `suite_key`, `status`, feature names, source paths, class names, test names, and similar object keys appear again and again. JSON stores all of that as repeated text and nested objects. Even if HTTP compression helps over the network, the browser still has to parse it into JavaScript objects, and Git still stores changed blobs. This kind of data is much closer to a table than a document, so a columnar format makes more sense.
+
+3. Finally, the search index data was not normalized correctly. The index repeated run metadata and case metadata too much, and it grew quickly. For Git history, this is especially bad because every new run changes a large generated file again.
+
+So then I basically applied three optimizations: lazy loading, Parquet compression, and search/index normalization.
+
+1. **Lazy loading.** The page now loads the light catalog/index first. Full run details are fetched only when you open the latest run or expand an archived run. Search index loading is also delayed until you actually search, and the browser can keep the FlexSearch index in IndexedDB. Logs are fetched only when you open a log modal. Source snippets are fetched only when you open a case. This makes the first page load much cheaper.
+2. **Compress data as Parquet.** Instead of treating every run as one large JSON document, the report data is split into Parquet tables: catalog runs, suites, features, per-run metadata, per-run cases, search rows, log files, and log lines. The Parquet writer uses Zstd compression and dictionary encoding, which is a much better fit for repeated values like suite names, statuses, feature tags, and source paths. The frontend then queries these Parquet files in the browser through DuckDB-Wasm.
+3. **Index file optimization.** The search rows now contain normalized fields and a short `detail_preview`, while the full case detail can be hydrated from the per-run case Parquet file when needed. For JSON fallback, the index and search data are partitioned into smaller shards. For the normal Parquet path, the global search index is a compressed Parquet file. The search index also has a stable `index_id`, so the browser can reuse the IndexedDB cache when the data has not changed.
+
+Result is crazy:
+
+1. Parquet data now:
+ 1. search rows: 7.64 MiB, 33,876 rows, growing about 0.142-0.147 MiB/run
+ 2. run case/detail data excluding search: 4.92 MiB across 52 runs, growing about 0.093 MiB/run
+ 3. catalog/index data: 0.03 MiB now, growing less than 1 KiB/run
+ 4. total Pages data: 12.59 MiB across 52 runs
+ 5. latest run data: 246,696 bytes \= 0.235 MiB
+2. Projection at the current rate:
+ 1. about \+0.236 MiB/run
+ 2. about \+7.1 MiB/month
+ 3. about \+86 MiB/year
+2. Compared with the JSON baseline:
+ 1. current data size: 149.11 MiB \-\> 12.59 MiB, about 11.8x smaller
+ 2. yearly growth: \~1.0 GiB/year \-\> \~86 MiB/year, about 12x slower
+ 3. search data: 107.09 MiB \-\> 7.64 MiB, about 14x smaller
+
+## Current Effect
+
+Ozone has already improved since the dashboard was released.
+
+
+I don’t want to overclaim that every improvement below was caused only by this dashboard, but I do think the dashboard changed the workflow. It gives us concrete failing cases, links that can be pasted into PRs, and an easy way to check whether the gap is still there after a patch.
+
+There are already quite a few PRs and designs which aim to fix compatibility gaps discovered or made easier to discuss by the dashboard:
+
+- Object tagging and bucket tagging:
+ - [HDDS-15258. Include x-amz-tagging-count header in HEAD object tagging responses](https://github.com/apache/ozone/pull/10265)
+ - [HDDS-15283. GetObjectTagging should return TagSet in sorted order of key](https://github.com/apache/ozone/pull/10277)
+ - [HDDS-15259. PutObject should treat null tag value for x-amz-tagging header as empty tag value](https://github.com/apache/ozone/pull/10299)
+ - [HDDS-15510. Implement OM read/write paths for bucket tagging with audit and metrics](https://github.com/apache/ozone/pull/10498)
+ - [Proposal: S3 Bucket Tagging in Apache Ozone](https://docs.google.com/document/d/1jq1JOatCrUpD7k3PutfkGrZ7nFC1C69BkMoMWcRWaw4/edit?tab=t.0#heading=h.rs6cprl5cddi)
+- Bucket listing and directory bucket behavior:
+ - [HDDS-15302. Throw S3 InvalidArgument for RequestParameters getInt if parsing error](https://github.com/apache/ozone/pull/10297)
+ - [HDDS-15303. Enable prefix filter for OBS](https://github.com/apache/ozone/pull/10298)
+ - [HDDS-15450. Implement ListDirectoryBuckets S3 API for Directory Buckets support](https://github.com/apache/ozone/pull/10408)
+- Other compatibility-related correctness work:
+ - [HDDS-15515. Support object Content-Type end-to-end in S3 Gateway](https://github.com/apache/ozone/pull/10472)
+
+The most useful effect is not just that the percentage moved. It is that the compatibility gap became much more concrete. A failed test can now become a dashboard link, then a JIRA, then a PR, then a nightly result showing whether it really passed.
+
+## Insight
+
+1. The main reason this really got adopted by other community members is that I posted it at the first time after I thought it could really work: [\[DISCUSS\] Nightly Ozone S3 compatibility report](https://lists.apache.org/thread/2td4jt8r26pwph5lwq3ncbnv97n27d5w). It was not perfect yet, but it was already useful enough for people to try. I think posting early was important.
Review Comment:
**Chinglish:** English doesn't use "at the first time."
**Suggested:** "I posted it the first time I thought it was ready to share" or "I shared it as soon as it was useful enough for others to try."
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
Review Comment:
**Suggested:** "An increasing number of database systems are moving storage to S3" (instead of "More database systems are moving their underlying storage").
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
Review Comment:
**Suggested:** "relying on external arbiters" (instead of "asking external arbiters").
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
+
+AWS also noted that conditional requests [let customers remove workaround code and simplify their systems](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=When%20we%20moved%20S3%20to,similar%20reaction). The same storage-level feature also [powers S3 Tables](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=they%20involve%20a%20complex,storage%2Dlevel%20features), which manages tabular data on S3.
+
+Common conditional request patterns include:
+
+- **Conditional `PUT`**
+ - Use `If-None-Match: *` to create an object only if it does not already exist.
+ - Use `If-Match: <etag>` to overwrite an object only if the current ETag is still the one the client observed.
+ - Use case: metadata commits, manifest updates, distributed lock files, and create-only log chunks.
+- **Conditional `GET`**
+ - Use `If-None-Match: <etag>` or date-based conditions to download the object only when it has changed.
+ - Use case: cache validation, dynamic model reloading, config refresh, and avoiding repeated downloads of large objects.
+- **Conditional `HEAD`**
+ - Same idea as conditional `GET`, but metadata-only.
+ - Use case: checking whether an object is still current without fetching the body.
+- **Conditional `POST` / `CompleteMultipartUpload`**
+ - Multipart upload writes data in parts, but the destination object becomes visible only when `CompleteMultipartUpload` succeeds.
+ - Conditional completion lets the client say: “complete this large object only if the destination still does not exist” or “only if the destination still matches this ETag.”
+ - Use case: large object writes where the final commit still needs create-only or compare-and-swap semantics.
+- **Conditional `COPY`**
+ - Source-side conditions make sure the source object is the version the client expects.
+ - Destination-side conditions make sure the copy does not overwrite unexpected data.
+ - Use case: snapshot promotion, clone/copy workflows, and metadata copy where both source and destination need validation.
+- **Conditional `DELETE`**
+ - Use `If-Match: <etag>` to delete only if the object is still unchanged.
+ - Use `If-Match: *` to delete only if the object exists.
+ - Use case: preventing a client from deleting an object that another writer has already replaced.
+
+For more advanced usage, let’s look at how modern systems use these APIs.
+
+## Application use cases
+
+So who exactly can use this?
Review Comment:
**Suggested:** "Who is this for?" or "Which workloads benefit?" (instead of "So who exactly can use this?").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Add S3-compat and S3 conditional request blog [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #488:
URL: https://github.com/apache/ozone-site/pull/488#discussion_r3574366564
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
Review Comment:
**Optional nit:** "backend-ish" reads informal.
**Suggested:** "the automation layer" or "the non-frontend parts."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
Review Comment:
**Optional nit:** "one-shotted" is informal/slang.
**Suggested:** "got most of it right on the first try" or "nailed the scaffold in one pass."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
Review Comment:
**Optional nit:** "laggy" is fine for informal tone; for a more formal ASF blog voice, consider "sluggish" or "slow."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
+
+1. First, we originally used one JSON file to represent one run result. Each latest run was about 0.236 MiB. This sounds small, but the problem is that we did not lazy load these files at the beginning. Every time the page refreshed, even if some files might already be cached by the browser, the TypeScript data structure still ended up holding too much content from too many files. That is very memory-consuming.
+
+2. Second, the JSON data had a ton of room to compress. The data shape is very repetitive: `run_id`, `suite_key`, `status`, feature names, source paths, class names, test names, and similar object keys appear again and again. JSON stores all of that as repeated text and nested objects. Even if HTTP compression helps over the network, the browser still has to parse it into JavaScript objects, and Git still stores changed blobs. This kind of data is much closer to a table than a document, so a columnar format makes more sense.
+
+3. Finally, the search index data was not normalized correctly. The index repeated run metadata and case metadata too much, and it grew quickly. For Git history, this is especially bad because every new run changes a large generated file again.
+
+So then I basically applied three optimizations: lazy loading, Parquet compression, and search/index normalization.
+
+1. **Lazy loading.** The page now loads the light catalog/index first. Full run details are fetched only when you open the latest run or expand an archived run. Search index loading is also delayed until you actually search, and the browser can keep the FlexSearch index in IndexedDB. Logs are fetched only when you open a log modal. Source snippets are fetched only when you open a case. This makes the first page load much cheaper.
+2. **Compress data as Parquet.** Instead of treating every run as one large JSON document, the report data is split into Parquet tables: catalog runs, suites, features, per-run metadata, per-run cases, search rows, log files, and log lines. The Parquet writer uses Zstd compression and dictionary encoding, which is a much better fit for repeated values like suite names, statuses, feature tags, and source paths. The frontend then queries these Parquet files in the browser through DuckDB-Wasm.
+3. **Index file optimization.** The search rows now contain normalized fields and a short `detail_preview`, while the full case detail can be hydrated from the per-run case Parquet file when needed. For JSON fallback, the index and search data are partitioned into smaller shards. For the normal Parquet path, the global search index is a compressed Parquet file. The search index also has a stable `index_id`, so the browser can reuse the IndexedDB cache when the data has not changed.
+
+Result is crazy:
+
+1. Parquet data now:
+ 1. search rows: 7.64 MiB, 33,876 rows, growing about 0.142-0.147 MiB/run
+ 2. run case/detail data excluding search: 4.92 MiB across 52 runs, growing about 0.093 MiB/run
+ 3. catalog/index data: 0.03 MiB now, growing less than 1 KiB/run
+ 4. total Pages data: 12.59 MiB across 52 runs
+ 5. latest run data: 246,696 bytes \= 0.235 MiB
+2. Projection at the current rate:
+ 1. about \+0.236 MiB/run
+ 2. about \+7.1 MiB/month
+ 3. about \+86 MiB/year
+2. Compared with the JSON baseline:
+ 1. current data size: 149.11 MiB \-\> 12.59 MiB, about 11.8x smaller
+ 2. yearly growth: \~1.0 GiB/year \-\> \~86 MiB/year, about 12x slower
+ 3. search data: 107.09 MiB \-\> 7.64 MiB, about 14x smaller
+
+## Current Effect
+
+Ozone has already improved since the dashboard was released.
+
+
+I don’t want to overclaim that every improvement below was caused only by this dashboard, but I do think the dashboard changed the workflow. It gives us concrete failing cases, links that can be pasted into PRs, and an easy way to check whether the gap is still there after a patch.
Review Comment:
**Optional nit:** "overclaim" is non-standard.
**Suggested:** "overstate" or "attribute every improvement solely to this dashboard."
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
+
+AWS also noted that conditional requests [let customers remove workaround code and simplify their systems](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=When%20we%20moved%20S3%20to,similar%20reaction). The same storage-level feature also [powers S3 Tables](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=they%20involve%20a%20complex,storage%2Dlevel%20features), which manages tabular data on S3.
+
+Common conditional request patterns include:
+
+- **Conditional `PUT`**
+ - Use `If-None-Match: *` to create an object only if it does not already exist.
+ - Use `If-Match: <etag>` to overwrite an object only if the current ETag is still the one the client observed.
+ - Use case: metadata commits, manifest updates, distributed lock files, and create-only log chunks.
+- **Conditional `GET`**
+ - Use `If-None-Match: <etag>` or date-based conditions to download the object only when it has changed.
+ - Use case: cache validation, dynamic model reloading, config refresh, and avoiding repeated downloads of large objects.
+- **Conditional `HEAD`**
+ - Same idea as conditional `GET`, but metadata-only.
+ - Use case: checking whether an object is still current without fetching the body.
+- **Conditional `POST` / `CompleteMultipartUpload`**
+ - Multipart upload writes data in parts, but the destination object becomes visible only when `CompleteMultipartUpload` succeeds.
+ - Conditional completion lets the client say: “complete this large object only if the destination still does not exist” or “only if the destination still matches this ETag.”
+ - Use case: large object writes where the final commit still needs create-only or compare-and-swap semantics.
+- **Conditional `COPY`**
+ - Source-side conditions make sure the source object is the version the client expects.
+ - Destination-side conditions make sure the copy does not overwrite unexpected data.
+ - Use case: snapshot promotion, clone/copy workflows, and metadata copy where both source and destination need validation.
+- **Conditional `DELETE`**
+ - Use `If-Match: <etag>` to delete only if the object is still unchanged.
+ - Use `If-Match: *` to delete only if the object exists.
+ - Use case: preventing a client from deleting an object that another writer has already replaced.
+
+For more advanced usage, let’s look at how modern systems use these APIs.
Review Comment:
**Optional nit:**
**Suggested:** "To see this in practice, let's look at how modern systems use these APIs."
##########
blog/2026-06-27-ozone-s3-compatibility/index.md:
##########
@@ -0,0 +1,164 @@
+---
+title: "Making Apache Ozone’s S3 Compatibility Visible"
+date: 2026-06-27
+authors: ["peterxcli"]
+tags: [Ozone, S3, compatibility]
+---
+
+<!-- cspell:ignore peterxcli Xiangpeng ish laggy Zstd overclaim amz -->
+
+Recently I created [https://ozone.s3.peterxcli.dev/](https://ozone.s3.peterxcli.dev/), an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR really helps the compatibility story. It is fully open source at [https://github.com/peterxcli/ozone-s3-compatibility](https://github.com/peterxcli/ozone-s3-compatibility).
+
+<!-- truncate -->
+
+TL;DR:
+
+- This really helped me get promoted as a PMC member of the Apache Ozone project.
+- Overall, **7% (50/711)** compatibility improvement at the time that this article was written.
+- Parquet \+ search-index optimization made the Pages data **11.8x smaller**, yearly git growth **12x slower**, and search data **14x smaller**.
+- It runs Apache Ozone against [`ceph/s3-tests`](https://github.com/ceph/s3-tests) and [`minio/mint`](https://github.com/minio/mint), then publishes a GitHub Pages report.
+- It turns compatibility from “a few epic tickets and some memory in people’s heads” into a searchable, linkable, daily signal.
+- For users, it answers “does Ozone support this S3 behavior today?” For developers, it answers “which exact test is failing, why, and did my PR move it?”
+
+## Why did I create it?
+
+Before this dashboard was introduced, Apache Ozone already tried really hard to track the compatibility gap with a few epic issues, for example: [Ozone S3 gateway Phase 4](https://issues.apache.org/jira/browse/HDDS-1186), [Ozone S3 gateway (phase III)](https://issues.apache.org/jira/browse/HDDS-12716), and [S3 API compatibility improvements and fixes](https://issues.apache.org/jira/browse/HDDS-8423).
+
+But I believe if you really click into the above links, you will quickly get lost among them. They are useful, but they are not a fresh report of what Apache Ozone’s S3 compatibility looks like today.
+
+Users don’t know whether Ozone really supports a specific S3 behavior. Developers don’t know whether a feature gap, implementation gap, or corner case is already handled correctly. And when a compatibility issue is fixed or newly introduced, there was no fresh source of truth to show that change.
+
+Also, when we develop S3 features, we often look at AWS S3 docs or source code from other open source S3-compatible systems, then infer how the implementation should behave and what test cases we should add. That works sometimes, but it is still mostly guessing. There was no easy harness to continuously test Ozone’s behavior against existing compatibility suites and make the result easy to inspect.
+
+So that’s why I wanted to create this dashboard.
+
+## How to use it?
+
+I usually think of the dashboard as having a few different modes.
+
+First, start from the latest run summary. The top cards show the current `s3-tests` and `mint` compatibility rate, eligible case count, pass/fail/error count, skipped count, and the delta compared with the previous run. This makes it easy to know whether today’s result is better, worse, or basically unchanged. One small detail here: the compatibility rate is `passed / (passed + failed + errored)`.
+
+
+
+Second, use the search box as the main debugging entry point. You can search by suite, test name, run id, run date, source path, feature name, status, or failure text. For example, if you are working on object tagging, you can search for `tagging`; if you are looking at a specific failure from `s3-tests`, you can paste part of the error message. Search results show the matched fields, suite, run date, status, feature tags, and a short failure preview.
+
+
+Third, click a search result. The modal shows the full failure detail, related metadata, a permalink, and the test code. For `s3-tests`, the dashboard can fetch the upstream source file and extract the Python test function, so you don’t need to jump between the report, GitHub, and your local checkout just to understand what the test is really asking for.
+
+
+
+Fourth, open the current report detail. The page does not force-load all case details at the beginning. Instead, it loads summary first, then fetches the full run detail when you open the latest run or an archived run. Inside the run detail, you can inspect suites, feature summaries, individual failed cases, and log files.
+
+
+
+
+Fifth, use the trend panel. The top-line trend is useful to see whether the overall compatibility is moving. The per-feature trend is even more useful because a single global number can hide what really changed. For example, a PR might only improve object tagging, bucket listing, copy object behavior, or some conditional-write corner case. Those changes are much easier to reason about feature by feature.
+
+
+
+Sixth, use archived runs when you need history. Old runs are still there, but their details are loaded only when opened. This is helpful when you want to compare a failure today with an older run, or when a PR says “this test used to fail before the fix.”
+
+
+Finally, the Parquet file inspector is there for people who want to inspect the published dataset itself. This is mostly for debugging the dashboard and data model, but it is surprisingly useful when you want to understand what files were generated, how many rows they contain, and whether the published data looks sane. Also thanks Xiangpeng for building the [Parquet Viewer](https://github.com/XiangpengHao/parquet-viewer), which let us inspect the parquet file content directly on the site.
+
+
+
+For PR authors, there is also a PR comparison workflow. It can run an Ozone PR head, compare it with the latest published main run, write a step summary, and optionally comment back on the Ozone PR. That is the part I like the most: compatibility is not just a dashboard to look at, but something you can wire into development review.
+
+## How is this built?
+
+For the very first version, I literally just prompted Codex with `gpt5.4 xhigh`:
+
+:::info Prompt
+I want to create a repo that runs [https://github.com/ceph/s3-tests](https://github.com/ceph/s3-tests) and [https://github.com/minio/mint](https://github.com/minio/mint) on GitHub Action and generate the compatibility report page on GitHub Pages nightly. Make the page pretty. My desired steps that run in the GitHub Action are: clone the Ozone repo, pull latest master change, compile, start running cluster, run Mint and s3-tests to get result, compile result to be the page. We can show the compatibility rate of each feature for each test (`s3-tests`, `mint`) daily change as a chart at the page top, followed by the report page, then also include the old result expansion button to allow users to check a specific date’s running result. BTW I have the local clone of Ozone at `~/Documents/oss/apache/ozone`, you can look into it directly without searching the codebase on the website.
+:::
+
+Then it almost one-shotted this. Actually I still made some follow-ups on the result, like “attach `s3-tests` and `mint` as submodule” and “use `act` to test the GitHub Action really works locally within Docker”, but those were relatively minor compared with the first scaffold.
+
+The nightly workflow now does roughly this:
+
+1. Build the Vue/Vite report frontend.
+2. Clone Ozone and the upstream compatibility suites.
+3. Build an Ozone distribution.
+4. Start a packaged Ozone compose cluster.
+5. Run `s3-tests` and Mint against the S3 Gateway.
+6. Normalize the raw output into a run result.
+7. Build the static Pages output, including the historical data.
+8. Publish to `gh-pages` for scheduled runs.
+
+The frontend is Vue 3 \+ Vite. The backend-ish part is just scripts and GitHub Actions. The data processing is mostly Python. For the newer version, the published report data is JSON.
+
+### Optimization
+
+After a few runs, I started to notice that the dashboard was becoming laggy. After some inspection, I found a few reasons.
+
+1. First, we originally used one JSON file to represent one run result. Each latest run was about 0.236 MiB. This sounds small, but the problem is that we did not lazy load these files at the beginning. Every time the page refreshed, even if some files might already be cached by the browser, the TypeScript data structure still ended up holding too much content from too many files. That is very memory-consuming.
+
+2. Second, the JSON data had a ton of room to compress. The data shape is very repetitive: `run_id`, `suite_key`, `status`, feature names, source paths, class names, test names, and similar object keys appear again and again. JSON stores all of that as repeated text and nested objects. Even if HTTP compression helps over the network, the browser still has to parse it into JavaScript objects, and Git still stores changed blobs. This kind of data is much closer to a table than a document, so a columnar format makes more sense.
+
+3. Finally, the search index data was not normalized correctly. The index repeated run metadata and case metadata too much, and it grew quickly. For Git history, this is especially bad because every new run changes a large generated file again.
+
+So then I basically applied three optimizations: lazy loading, Parquet compression, and search/index normalization.
+
+1. **Lazy loading.** The page now loads the light catalog/index first. Full run details are fetched only when you open the latest run or expand an archived run. Search index loading is also delayed until you actually search, and the browser can keep the FlexSearch index in IndexedDB. Logs are fetched only when you open a log modal. Source snippets are fetched only when you open a case. This makes the first page load much cheaper.
+2. **Compress data as Parquet.** Instead of treating every run as one large JSON document, the report data is split into Parquet tables: catalog runs, suites, features, per-run metadata, per-run cases, search rows, log files, and log lines. The Parquet writer uses Zstd compression and dictionary encoding, which is a much better fit for repeated values like suite names, statuses, feature tags, and source paths. The frontend then queries these Parquet files in the browser through DuckDB-Wasm.
+3. **Index file optimization.** The search rows now contain normalized fields and a short `detail_preview`, while the full case detail can be hydrated from the per-run case Parquet file when needed. For JSON fallback, the index and search data are partitioned into smaller shards. For the normal Parquet path, the global search index is a compressed Parquet file. The search index also has a stable `index_id`, so the browser can reuse the IndexedDB cache when the data has not changed.
+
+Result is crazy:
+
+1. Parquet data now:
+ 1. search rows: 7.64 MiB, 33,876 rows, growing about 0.142-0.147 MiB/run
+ 2. run case/detail data excluding search: 4.92 MiB across 52 runs, growing about 0.093 MiB/run
+ 3. catalog/index data: 0.03 MiB now, growing less than 1 KiB/run
+ 4. total Pages data: 12.59 MiB across 52 runs
+ 5. latest run data: 246,696 bytes \= 0.235 MiB
+2. Projection at the current rate:
+ 1. about \+0.236 MiB/run
+ 2. about \+7.1 MiB/month
+ 3. about \+86 MiB/year
+2. Compared with the JSON baseline:
+ 1. current data size: 149.11 MiB \-\> 12.59 MiB, about 11.8x smaller
+ 2. yearly growth: \~1.0 GiB/year \-\> \~86 MiB/year, about 12x slower
+ 3. search data: 107.09 MiB \-\> 7.64 MiB, about 14x smaller
+
+## Current Effect
+
+Ozone has already improved since the dashboard was released.
+
+
+I don’t want to overclaim that every improvement below was caused only by this dashboard, but I do think the dashboard changed the workflow. It gives us concrete failing cases, links that can be pasted into PRs, and an easy way to check whether the gap is still there after a patch.
+
+There are already quite a few PRs and designs which aim to fix compatibility gaps discovered or made easier to discuss by the dashboard:
+
+- Object tagging and bucket tagging:
+ - [HDDS-15258. Include x-amz-tagging-count header in HEAD object tagging responses](https://github.com/apache/ozone/pull/10265)
+ - [HDDS-15283. GetObjectTagging should return TagSet in sorted order of key](https://github.com/apache/ozone/pull/10277)
+ - [HDDS-15259. PutObject should treat null tag value for x-amz-tagging header as empty tag value](https://github.com/apache/ozone/pull/10299)
+ - [HDDS-15510. Implement OM read/write paths for bucket tagging with audit and metrics](https://github.com/apache/ozone/pull/10498)
+ - [Proposal: S3 Bucket Tagging in Apache Ozone](https://docs.google.com/document/d/1jq1JOatCrUpD7k3PutfkGrZ7nFC1C69BkMoMWcRWaw4/edit?tab=t.0#heading=h.rs6cprl5cddi)
+- Bucket listing and directory bucket behavior:
+ - [HDDS-15302. Throw S3 InvalidArgument for RequestParameters getInt if parsing error](https://github.com/apache/ozone/pull/10297)
+ - [HDDS-15303. Enable prefix filter for OBS](https://github.com/apache/ozone/pull/10298)
+ - [HDDS-15450. Implement ListDirectoryBuckets S3 API for Directory Buckets support](https://github.com/apache/ozone/pull/10408)
+- Other compatibility-related correctness work:
+ - [HDDS-15515. Support object Content-Type end-to-end in S3 Gateway](https://github.com/apache/ozone/pull/10472)
+
+The most useful effect is not just that the percentage moved. It is that the compatibility gap became much more concrete. A failed test can now become a dashboard link, then a JIRA, then a PR, then a nightly result showing whether it really passed.
+
+## Insight
+
+1. The main reason this really got adopted by other community members is that I posted it at the first time after I thought it could really work: [\[DISCUSS\] Nightly Ozone S3 compatibility report](https://lists.apache.org/thread/2td4jt8r26pwph5lwq3ncbnv97n27d5w). It was not perfect yet, but it was already useful enough for people to try. I think posting early was important.
+2. This also really helped me get promoted as a PMC member of the Apache Ozone project. Not because the dashboard itself is magical, but because it made a long-standing problem visible and gave the community a practical tool to keep improving it.
Review Comment:
**Optional nit:** This repeats the PMC-promotion point from the TL;DR (line 16). Consider keeping only one mention unless both are intentional.
##########
blog/2026-07-08-ozone-s3-conditional-request/index.md:
##########
@@ -0,0 +1,417 @@
+---
+title: "Fast Optimistic Concurrency Control with Apache Ozone S3 Conditional Requests"
+date: 2026-07-08
+authors: ["peterxcli"]
+tags: [Ozone, S3, concurrency, conditional-requests]
+---
+
+<!-- cspell:ignore peterxcli RDBMS LSN CAS amaliujia Turbopuffer MVCC -->
+
+:::note
+Apache Ozone will support conditional `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, and `CompleteMultipartUpload` in the upcoming [2.2 release](https://github.com/apache/ozone/releases/tag/ozone-2.2.0-RC0) (RC0 is currently under a [vote](https://lists.apache.org/thread/gz567ljydh4ht63h6c9pjfclrbrrr9z7)), and will add conditional `DeleteObject` and `DeleteObjects` support in 2.3.
+:::
+
+More database systems are moving their underlying storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.
+
+Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.
+
+Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.
+
+<!-- truncate -->
+
+## TL;DR
+
+Apache Ozone is adding S3 conditional request support for operations such as `PutObject`, `GetObject`, `HeadObject`, `CopyObject`, `DeleteObject`, and `CompleteMultipartUpload`. This allows applications to perform an operation only when the object state matches expected conditions, such as “create this object only if it does not already exist” or “overwrite this object only if the current ETag still matches.”
+
+Under the hood, Ozone reuses its atomic rewrite path. For conditional writes, the S3 Gateway passes the caller’s expected ETag to Ozone Manager (OM). OM validates it near the metadata write path, stores the matched generation in the open key, and revalidates it during commit. This keeps the ETag check close to the place where the object state changes, so Ozone can provide a distributed CAS-style primitive without adding an extra gateway-side metadata read on the successful path.
+
+This unlocks safer optimistic concurrency control for data systems built on top of object storage, including metadata catalogs, WAL-like workflows, leader-election patterns, queues, model-serving caches, and object-store-backed databases.
+
+## What is an S3 conditional request?
+
+:::note
+You can use conditional requests to add preconditions to your S3 operations. To use conditional requests, you add an additional header to your Amazon S3 API operation. This header specifies a condition that, if not met, will result in the S3 operation failing.
+
+Source: [Amazon S3 conditional requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html)
+:::
+
+Conditional requests allow atomic CAS operations on target objects: clients can coordinate through S3 objects instead of asking external arbiters. Using S3 conditional requests is like moving part of the coordination logic into storage.
+
+AWS also noted that conditional requests [let customers remove workaround code and simplify their systems](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=When%20we%20moved%20S3%20to,similar%20reaction). The same storage-level feature also [powers S3 Tables](https://www.allthingsdistributed.com/2025/03/in-s3-simplicity-is-table-stakes.html#:~:text=they%20involve%20a%20complex,storage%2Dlevel%20features), which manages tabular data on S3.
+
+Common conditional request patterns include:
+
+- **Conditional `PUT`**
+ - Use `If-None-Match: *` to create an object only if it does not already exist.
+ - Use `If-Match: <etag>` to overwrite an object only if the current ETag is still the one the client observed.
+ - Use case: metadata commits, manifest updates, distributed lock files, and create-only log chunks.
+- **Conditional `GET`**
+ - Use `If-None-Match: <etag>` or date-based conditions to download the object only when it has changed.
+ - Use case: cache validation, dynamic model reloading, config refresh, and avoiding repeated downloads of large objects.
+- **Conditional `HEAD`**
+ - Same idea as conditional `GET`, but metadata-only.
+ - Use case: checking whether an object is still current without fetching the body.
+- **Conditional `POST` / `CompleteMultipartUpload`**
+ - Multipart upload writes data in parts, but the destination object becomes visible only when `CompleteMultipartUpload` succeeds.
+ - Conditional completion lets the client say: “complete this large object only if the destination still does not exist” or “only if the destination still matches this ETag.”
+ - Use case: large object writes where the final commit still needs create-only or compare-and-swap semantics.
+- **Conditional `COPY`**
+ - Source-side conditions make sure the source object is the version the client expects.
+ - Destination-side conditions make sure the copy does not overwrite unexpected data.
+ - Use case: snapshot promotion, clone/copy workflows, and metadata copy where both source and destination need validation.
+- **Conditional `DELETE`**
+ - Use `If-Match: <etag>` to delete only if the object is still unchanged.
+ - Use `If-Match: *` to delete only if the object exists.
+ - Use case: preventing a client from deleting an object that another writer has already replaced.
+
+For more advanced usage, let’s look at how modern systems use these APIs.
+
+## Application use cases
+
+So who exactly can use this?
+
+### Dynamic AI model reloading
+
+Advanced inference servers often cache massive model weights on local NVMe drives. To keep these models updated without constantly re-downloading them every few minutes, they can use S3 conditional requests. For example, the inference server can send a `GET` request with `If-None-Match` and the ETag of the model it already has.
+
+If the ETag still matches, Ozone can return `304 Not Modified`, so the server keeps using the local cached model and skips the download. If the ETag does not match, the model has been updated, so the server downloads the newest model and stores the new ETag for the next check. Thanks to [@amaliujia](https://github.com/amaliujia) for the example.
+
+
+
+*Figure 1. An inference server uses `If-None-Match` to avoid re-downloading unchanged model weights.*
+
+### Turbopuffer
+
+Turbopuffer’s object-storage queue is a great example of using a small metadata object to coordinate larger data work. In the simple version, a pusher reads `queue.json`, appends a job, and writes it back with CAS. A worker does the same thing to claim the next job. The CAS write only succeeds if `queue.json` has not changed since the client read it; otherwise the client reads the new queue and retries.
+
+This design is simple, but many clients can contend on one queue object. Turbopuffer improves it by batching updates with group commit, then moving the object-storage interaction behind a stateless broker. Clients talk to the broker, the broker performs one group-commit loop, and it only acknowledges work after the updated queue is durably committed.
+
+
+
+*Figure 2. Turbopuffer’s object-storage queue batches contended CAS updates behind a broker. Source: [Turbopuffer object storage queue](https://turbopuffer.com/blog/object-storage-queue).*
+
+### SlateDB
+
+SlateDB is an embedded LSM-tree storage engine built for object storage. It stores durable state as object-store files: WAL SSTs, compacted SSTs, and manifest files. Like other LSM engines, it relies on immutable files for most data movement, but it still needs a safe coordination point to decide which writer owns the next part of the log.
+
+Conditional object creation is useful here because SlateDB can use object names as fencing points. Instead of overwriting a shared file, the writer creates the next manifest or WAL object with `If-None-Match: *`. The write succeeds only if that object does not already exist.
+
+
+
+*Figure 3. A SlateDB writer uses conditional object creation as a fencing point for object-store state.*
+
+### Iceberg file catalog
+
+Iceberg is another natural fit for this style of coordination. Iceberg table state is maintained in metadata files, and each commit creates a new metadata file. The commit succeeds by atomically swapping the table metadata pointer from the old metadata file to the new one. If another writer commits first, the swap fails, and the writer retries against the new table state.
+
+For an Iceberg-style file catalog on object storage, conditional requests can provide the missing compare-and-swap primitive. The catalog can store a small pointer object, such as `current.json`, that points to the latest table metadata file. A writer first writes the new metadata file using a unique name, then updates the current metadata pointer only if its ETag still matches the ETag that the writer read.
+
+
+
+*Figure 4. An Iceberg-style file catalog updates a small metadata pointer only if its ETag still matches.*
+
+### Leader election
+
+Leader election can also be built on top of conditional writes. The basic idea is to have all nodes compete to create the next lock file, such as `lock-0000000002.json`, using `If-None-Match: *`. Only one node can create the file successfully. That node becomes the leader for that epoch; the others receive a precondition failure and keep watching.
+
+
+
+*Figure 5. Nodes compete to create the next epoch lock file; only one writer succeeds.*
+
+Leader election by itself is not enough. A paused old leader can come back and still believe it owns the lock. This is the “zombie leader” problem. The fix is to use the leader epoch as a fencing token. Every request made by the leader includes the epoch, and downstream systems reject requests with an older epoch than the highest one they have already seen.
+
+
+
+*Figure 6. Epoch fencing prevents an old paused leader from writing after a newer leader takes over.*
+
+The leader should periodically update the lock file it acquired to signal liveness. Other nodes can poll the lock and check whether the lock was released or expired by looking at `Last-Modified`, which S3 exposes as standard object metadata.
+
+### WAL write/get with OSWALD
+
+OSWALD, the Object Storage Write-Ahead Log Device, shows how to build a WAL directly on object storage primitives. The design has three object types: a manifest object, snapshots, and log chunks. The manifest tracks the latest checkpoint and garbage collection progress; chunks hold the log content.
+
+Appending to the WAL can be done with conditional object creation. A writer creates the next chunk with `PUT If-None-Match`. If another writer already created that chunk for the same LSN, the write fails and the writer catches up by tailing the log. After creating a chunk, the writer also checks the manifest with `GET If-None-Match` to make sure garbage collection has not moved past its LSN before acknowledging the write.
+
+
+
+*Figure 7. OSWALD uses manifest, snapshot, and chunk objects to build a WAL on object storage. Source: [OSWALD](https://github.com/nvartolomei/oswald).*
+
+Now let’s look at how Ozone keeps S3 conditional requests—the CAS primitive—fast.
Review Comment:
**Optional nit:**
**Suggested:** "Now let's look at how Ozone implements S3 conditional requests efficiently." (instead of "keeps … fast").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Add S3-compat and S3 conditional request blog [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #488:
URL: https://github.com/apache/ozone-site/pull/488#issuecomment-4962994793
**Optional nit (PR metadata):**
- Title: `HDDS-15801. Add S3-compat and S3 conditional request blog` → `HDDS-15801. Add blogs on S3 compatibility and S3 conditional requests`
- Description: `Add S3-compat and S3 conditional request blog` → `Add blogs on S3 compatibility and S3 conditional requests`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15801. Add blogs on S3 compatibility and S3 conditional requests [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #488:
URL: https://github.com/apache/ozone-site/pull/488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org