You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by xu...@apache.org on 2026/09/20 17:43:20 UTC
(opendal-yinyang) branch main updated: feat: add authenticated content preparation (#44)
This is an automated email from the ASF dual-hosted git repository.
Xuanwo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opendal-yinyang.git
The following commit(s) were added to refs/heads/main by this push:
new 99c937e feat: add authenticated content preparation (#44)
99c937e is described below
commit 99c937ec9a4486f71dff6f2e69bd5d4462a46d6d
Author: Xuanwo <gi...@xuanwo.io>
AuthorDate: Mon Sep 21 01:43:15 2026 +0800
feat: add authenticated content preparation (#44)
---
.github/workflows/ci.yml | 2 -
Cargo.lock | 1206 +++++++++++++++++++++++++++++++++++++++-
Cargo.toml | 1 +
crates/core/Cargo.toml | 9 +-
crates/core/src/content.rs | 197 +++++++
crates/core/src/data.rs | 782 ++++++++++++++++++++++++++
crates/core/src/error.rs | 5 +
crates/core/src/lib.rs | 2 +
crates/core/src/publication.rs | 4 +-
crates/core/tests/core.rs | 534 +++++++++++++++++-
specs/README.md | 1 +
specs/content-profile.md | 90 +++
specs/yinyang-format.md | 29 +
13 files changed, 2846 insertions(+), 16 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 3547ac7..27dd4a3 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -19,8 +19,6 @@ name: Apache OpenDAL™ YinYang CI
on:
pull_request:
- branches:
- - main
permissions:
contents: read
diff --git a/Cargo.lock b/Cargo.lock
index 21bd158..5b54d65 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -38,7 +38,7 @@ version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
- "windows-sys",
+ "windows-sys 0.61.2",
]
[[package]]
@@ -49,7 +49,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
- "windows-sys",
+ "windows-sys 0.61.2",
]
[[package]]
@@ -70,10 +70,45 @@ version = "0.6.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94a214ba60d6231afd0e805e3c27c45a1626d9debaa5a5061c45a1ea1b2f1ed0"
dependencies = [
- "hashbrown",
+ "hashbrown 0.17.1",
"slab",
]
+[[package]]
+name = "atomic-waker"
+version = "1.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
+
+[[package]]
+name = "aws-lc-rs"
+version = "1.18.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e"
+dependencies = [
+ "aws-lc-sys",
+ "zeroize",
+]
+
+[[package]]
+name = "aws-lc-sys"
+version = "0.45.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27"
+dependencies = [
+ "cc",
+ "cmake",
+ "dunce",
+ "fs_extra",
+ "pkg-config",
+]
+
+[[package]]
+name = "base64"
+version = "0.22.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
+
[[package]]
name = "base64"
version = "0.23.1"
@@ -86,6 +121,12 @@ version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
+[[package]]
+name = "bitflags"
+version = "2.13.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06"
+
[[package]]
name = "blake3"
version = "1.8.7"
@@ -137,6 +178,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ad534f4357a5264cce5019c989cf66a4f0dc4e0d1b1d15f8aacec0ff7360273"
dependencies = [
"find-msvc-tools",
+ "jobserver",
+ "libc",
"shlex",
]
@@ -152,6 +195,17 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
+[[package]]
+name = "chacha20"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "rand_core",
+]
+
[[package]]
name = "clap"
version = "4.6.6"
@@ -192,24 +246,85 @@ version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
+[[package]]
+name = "cmake"
+version = "0.1.58"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
+dependencies = [
+ "cc",
+]
+
+[[package]]
+name = "cmov"
+version = "0.5.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
+
[[package]]
name = "colorchoice"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"
+[[package]]
+name = "combine"
+version = "4.6.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cfc320937d09e6de266b31b9afb480f197d7a861be86be7cb2ea7e5d1bfffc5e"
+dependencies = [
+ "bytes",
+ "memchr",
+]
+
[[package]]
name = "const-oid"
version = "0.10.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
+[[package]]
+name = "const-random"
+version = "0.1.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359"
+dependencies = [
+ "const-random-macro",
+]
+
+[[package]]
+name = "const-random-macro"
+version = "0.1.16"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e"
+dependencies = [
+ "getrandom 0.2.17",
+ "once_cell",
+ "tiny-keccak",
+]
+
[[package]]
name = "constant_time_eq"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
+[[package]]
+name = "core-foundation"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
+dependencies = [
+ "core-foundation-sys",
+ "libc",
+]
+
+[[package]]
+name = "core-foundation-sys"
+version = "0.8.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
+
[[package]]
name = "cpufeatures"
version = "0.3.1"
@@ -219,6 +334,32 @@ dependencies = [
"libc",
]
+[[package]]
+name = "crc-fast"
+version = "1.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5"
+dependencies = [
+ "digest 0.10.7",
+ "spin",
+]
+
+[[package]]
+name = "crunchy"
+version = "0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
+
+[[package]]
+name = "crypto-common"
+version = "0.1.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
+dependencies = [
+ "generic-array",
+ "typenum",
+]
+
[[package]]
name = "crypto-common"
version = "0.2.2"
@@ -228,13 +369,22 @@ dependencies = [
"hybrid-array",
]
+[[package]]
+name = "ctutils"
+version = "0.4.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"
+dependencies = [
+ "cmov",
+]
+
[[package]]
name = "defmt"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1"
dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
"defmt-macros",
]
@@ -259,6 +409,15 @@ dependencies = [
"thiserror",
]
+[[package]]
+name = "digest"
+version = "0.10.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
+dependencies = [
+ "crypto-common 0.1.7",
+]
+
[[package]]
name = "digest"
version = "0.11.3"
@@ -267,7 +426,8 @@ checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
dependencies = [
"block-buffer",
"const-oid",
- "crypto-common",
+ "crypto-common 0.2.2",
+ "ctutils",
]
[[package]]
@@ -281,6 +441,21 @@ dependencies = [
"syn 3.0.3",
]
+[[package]]
+name = "dlv-list"
+version = "0.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f"
+dependencies = [
+ "const-random",
+]
+
+[[package]]
+name = "dunce"
+version = "1.0.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813"
+
[[package]]
name = "find-msvc-tools"
version = "0.1.11"
@@ -296,6 +471,12 @@ dependencies = [
"percent-encoding",
]
+[[package]]
+name = "fs_extra"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
+
[[package]]
name = "futures"
version = "0.3.34"
@@ -304,6 +485,7 @@ checksum = "9a31d2a3fbaaeb2af2368bbdd904aa8e812d3c04a1ee10d3171f52d556e5d0a3"
dependencies = [
"futures-channel",
"futures-core",
+ "futures-executor",
"futures-io",
"futures-sink",
"futures-task",
@@ -326,6 +508,17 @@ version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
+[[package]]
+name = "futures-executor"
+version = "0.3.34"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "031b47cf1a3c6cc8bc2fc76cd437f521619387907d469316e7c0bc278f1f5432"
+dependencies = [
+ "futures-core",
+ "futures-task",
+ "futures-util",
+]
+
[[package]]
name = "futures-io"
version = "0.3.34"
@@ -372,6 +565,29 @@ dependencies = [
"slab",
]
+[[package]]
+name = "generic-array"
+version = "0.14.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
+dependencies = [
+ "typenum",
+ "version_check",
+]
+
+[[package]]
+name = "getrandom"
+version = "0.2.17"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
+dependencies = [
+ "cfg-if",
+ "js-sys",
+ "libc",
+ "wasi",
+ "wasm-bindgen",
+]
+
[[package]]
name = "getrandom"
version = "0.4.3"
@@ -379,10 +595,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
dependencies = [
"cfg-if",
+ "js-sys",
"libc",
"r-efi",
+ "rand_core",
+ "wasm-bindgen",
]
+[[package]]
+name = "hashbrown"
+version = "0.14.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
+
[[package]]
name = "hashbrown"
version = "0.17.1"
@@ -395,6 +620,21 @@ version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
+[[package]]
+name = "hex"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
+
+[[package]]
+name = "hmac"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f"
+dependencies = [
+ "digest 0.11.3",
+]
+
[[package]]
name = "http"
version = "1.5.0"
@@ -405,6 +645,35 @@ dependencies = [
"itoa",
]
+[[package]]
+name = "http-body"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c"
+dependencies = [
+ "bytes",
+ "http",
+]
+
+[[package]]
+name = "http-body-util"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "23169fe34a5fbcdd3f3862e78fb9b6fccd5f02a6dc6f732547005d45631ce71c"
+dependencies = [
+ "bytes",
+ "futures-core",
+ "http",
+ "http-body",
+ "pin-project-lite",
+]
+
+[[package]]
+name = "httparse"
+version = "1.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
+
[[package]]
name = "hybrid-array"
version = "0.4.14"
@@ -414,6 +683,64 @@ dependencies = [
"typenum",
]
+[[package]]
+name = "hyper"
+version = "1.11.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "27b501faa50e7a26c3d3560ca625132f4078a17771f4810baf70475ae48cbe43"
+dependencies = [
+ "atomic-waker",
+ "bytes",
+ "futures-channel",
+ "futures-core",
+ "http",
+ "http-body",
+ "httparse",
+ "itoa",
+ "pin-project-lite",
+ "smallvec",
+ "tokio",
+ "want",
+]
+
+[[package]]
+name = "hyper-rustls"
+version = "0.27.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
+dependencies = [
+ "http",
+ "hyper",
+ "hyper-util",
+ "rustls",
+ "tokio",
+ "tokio-rustls",
+ "tower-service",
+]
+
+[[package]]
+name = "hyper-util"
+version = "0.1.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
+dependencies = [
+ "base64 0.22.1",
+ "bytes",
+ "futures-channel",
+ "futures-util",
+ "http",
+ "http-body",
+ "hyper",
+ "ipnet",
+ "libc",
+ "percent-encoding",
+ "pin-project-lite",
+ "socket2",
+ "tokio",
+ "tower-service",
+ "tracing",
+]
+
[[package]]
name = "icu_collections"
version = "2.3.0"
@@ -518,6 +845,12 @@ dependencies = [
"icu_properties",
]
+[[package]]
+name = "ipnet"
+version = "2.12.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
+
[[package]]
name = "is_terminal_polyfill"
version = "1.70.2"
@@ -585,6 +918,65 @@ dependencies = [
"jiff-tzdb",
]
+[[package]]
+name = "jni"
+version = "0.22.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498"
+dependencies = [
+ "cfg-if",
+ "combine",
+ "jni-macros",
+ "jni-sys",
+ "log",
+ "simd_cesu8",
+ "thiserror",
+ "walkdir",
+ "windows-link",
+]
+
+[[package]]
+name = "jni-macros"
+version = "0.22.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "rustc_version",
+ "simd_cesu8",
+ "syn 2.0.119",
+]
+
+[[package]]
+name = "jni-sys"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2"
+dependencies = [
+ "jni-sys-macros",
+]
+
+[[package]]
+name = "jni-sys-macros"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264"
+dependencies = [
+ "quote",
+ "syn 2.0.119",
+]
+
+[[package]]
+name = "jobserver"
+version = "0.1.35"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3"
+dependencies = [
+ "getrandom 0.4.3",
+ "libc",
+]
+
[[package]]
name = "js-sys"
version = "0.3.104"
@@ -614,6 +1006,12 @@ version = "0.4.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
+[[package]]
+name = "lru-slab"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4050469837a6ff301cd14c1f8f24f88549e6d548f24f64e2148eb0f72cebc51f"
+
[[package]]
name = "md-5"
version = "0.11.0"
@@ -621,7 +1019,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98"
dependencies = [
"cfg-if",
- "digest",
+ "digest 0.11.3",
+]
+
+[[package]]
+name = "mea"
+version = "0.6.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c709842c4ce65cb91e2666ad5319dfc1efc3af0d34f02075eddca9000d9f8afb"
+dependencies = [
+ "hashbrown 0.17.1",
+ "slab",
]
[[package]]
@@ -630,6 +1038,17 @@ version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
+[[package]]
+name = "mio"
+version = "1.2.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4b18443e9c262bfe8fa82f51666e2642c53393f7e5c27b3e1aeab922cff5b9d8"
+dependencies = [
+ "libc",
+ "wasi",
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "once_cell"
version = "1.21.4"
@@ -649,6 +1068,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "33dbff14cc9bb085224256d6a81289d2f3202e85b06f408d42534b42162a4231"
dependencies = [
"opendal-core",
+ "opendal-http-transport-reqwest",
+ "opendal-service-s3",
]
[[package]]
@@ -659,7 +1080,7 @@ checksum = "48dbcef97d3eb7591db2c18d5cae95c836bcce07359b98d98dd6f4e861eb77b7"
dependencies = [
"anyhow",
"asyncband",
- "base64",
+ "base64 0.23.1",
"bytes",
"futures",
"http",
@@ -668,6 +1089,7 @@ dependencies = [
"md-5",
"percent-encoding",
"quick-xml",
+ "reqsign-core",
"serde",
"serde_json",
"tokio",
@@ -676,6 +1098,57 @@ dependencies = [
"web-time",
]
+[[package]]
+name = "opendal-http-transport-reqwest"
+version = "0.58.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85663452ea32bbc17e8f79ab29788c846d116ec7de31451be9c787e462dcb36c"
+dependencies = [
+ "bytes",
+ "futures",
+ "http",
+ "http-body",
+ "opendal-core",
+ "reqwest",
+]
+
+[[package]]
+name = "opendal-service-s3"
+version = "0.58.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c64335f9f24ccb62ac36f1d976342b48611a75ba61979813a4f78a4ebd94de42"
+dependencies = [
+ "base64 0.23.1",
+ "bytes",
+ "crc-fast",
+ "http",
+ "log",
+ "md-5",
+ "opendal-core",
+ "quick-xml",
+ "reqsign-aws-v4",
+ "reqsign-core",
+ "reqsign-file-read-tokio",
+ "serde",
+ "url",
+]
+
+[[package]]
+name = "openssl-probe"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
+
+[[package]]
+name = "ordered-multimap"
+version = "0.7.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79"
+dependencies = [
+ "dlv-list",
+ "hashbrown 0.14.5",
+]
+
[[package]]
name = "percent-encoding"
version = "2.3.2"
@@ -688,6 +1161,12 @@ version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
+[[package]]
+name = "pkg-config"
+version = "0.3.34"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
+
[[package]]
name = "portable-atomic"
version = "1.15.0"
@@ -731,6 +1210,63 @@ dependencies = [
"serde",
]
+[[package]]
+name = "quinn"
+version = "0.11.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4051e23e9185c255a7e33ef59cdbca87a22d359052eecd22fc6b901fb37d9d11"
+dependencies = [
+ "bytes",
+ "cfg_aliases",
+ "pin-project-lite",
+ "quinn-proto",
+ "quinn-udp",
+ "rustc-hash",
+ "rustls",
+ "socket2",
+ "thiserror",
+ "tokio",
+ "tracing",
+ "web-time",
+]
+
+[[package]]
+name = "quinn-proto"
+version = "0.11.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
+dependencies = [
+ "aws-lc-rs",
+ "bytes",
+ "getrandom 0.4.3",
+ "lru-slab",
+ "rand",
+ "rand_pcg",
+ "ring",
+ "rustc-hash",
+ "rustls",
+ "rustls-pki-types",
+ "slab",
+ "thiserror",
+ "tinyvec",
+ "tracing",
+ "web-time",
+]
+
+[[package]]
+name = "quinn-udp"
+version = "0.5.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
+dependencies = [
+ "cfg_aliases",
+ "libc",
+ "once_cell",
+ "socket2",
+ "tracing",
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "quote"
version = "1.0.47"
@@ -746,12 +1282,312 @@ version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
+[[package]]
+name = "rand"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
+dependencies = [
+ "chacha20",
+ "getrandom 0.4.3",
+ "rand_core",
+]
+
+[[package]]
+name = "rand_core"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69"
+
+[[package]]
+name = "rand_pcg"
+version = "0.10.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a"
+dependencies = [
+ "rand_core",
+]
+
+[[package]]
+name = "reqsign-aws-core"
+version = "3.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bac4749b7dfa7bfaccd01eb03e9dc795ed37e3f20d6f0f38e2c67ee85ad6bc86"
+dependencies = [
+ "bytes",
+ "form_urlencoded",
+ "hex",
+ "http",
+ "log",
+ "percent-encoding",
+ "quick-xml",
+ "reqsign-core",
+ "rust-ini",
+ "serde",
+ "serde_json",
+ "serde_urlencoded",
+ "sha1",
+]
+
+[[package]]
+name = "reqsign-aws-v4"
+version = "3.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ff250f0fd0b913fbd565e405acc553da0f13bde30bfb5403178c9d0313cdc15f"
+dependencies = [
+ "bytes",
+ "http",
+ "log",
+ "quick-xml",
+ "reqsign-aws-core",
+ "reqsign-core",
+ "serde",
+]
+
+[[package]]
+name = "reqsign-core"
+version = "3.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ff052daffb0599681c50f85c59e7236438976efe991ab864edd9f3b235501a0f"
+dependencies = [
+ "anyhow",
+ "base64 0.23.1",
+ "bytes",
+ "futures",
+ "hex",
+ "hmac",
+ "http",
+ "jiff",
+ "log",
+ "mea",
+ "percent-encoding",
+ "sha1",
+ "sha2",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "reqsign-file-read-tokio"
+version = "3.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b3235df90a6bca681aa47dd86f2393d122a6d77042aa8a7c81e218cd45c5bfc0"
+dependencies = [
+ "anyhow",
+ "reqsign-core",
+ "tokio",
+]
+
+[[package]]
+name = "reqwest"
+version = "0.13.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "16a1cfa75cc186dd73d5818e510e042e40927bccc9c236b061cea97e1eb08029"
+dependencies = [
+ "base64 0.23.1",
+ "bytes",
+ "futures-core",
+ "futures-util",
+ "http",
+ "http-body",
+ "http-body-util",
+ "hyper",
+ "hyper-rustls",
+ "hyper-util",
+ "js-sys",
+ "log",
+ "percent-encoding",
+ "pin-project-lite",
+ "quinn",
+ "rustls",
+ "rustls-pki-types",
+ "rustls-platform-verifier",
+ "sync_wrapper",
+ "tokio",
+ "tokio-rustls",
+ "tokio-util",
+ "tower",
+ "tower-http",
+ "tower-service",
+ "url",
+ "wasm-bindgen",
+ "wasm-bindgen-futures",
+ "wasm-streams",
+ "web-sys",
+]
+
+[[package]]
+name = "ring"
+version = "0.17.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
+dependencies = [
+ "cc",
+ "cfg-if",
+ "getrandom 0.2.17",
+ "libc",
+ "untrusted",
+ "windows-sys 0.52.0",
+]
+
+[[package]]
+name = "rust-ini"
+version = "0.21.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7"
+dependencies = [
+ "cfg-if",
+ "ordered-multimap",
+]
+
+[[package]]
+name = "rustc-hash"
+version = "2.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
+
+[[package]]
+name = "rustc_version"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
+dependencies = [
+ "semver",
+]
+
+[[package]]
+name = "rustls"
+version = "0.23.45"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
+dependencies = [
+ "aws-lc-rs",
+ "once_cell",
+ "rustls-pki-types",
+ "rustls-webpki",
+ "subtle",
+ "zeroize",
+]
+
+[[package]]
+name = "rustls-native-certs"
+version = "0.8.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d"
+dependencies = [
+ "openssl-probe",
+ "rustls-pki-types",
+ "schannel",
+ "security-framework",
+]
+
+[[package]]
+name = "rustls-pki-types"
+version = "1.15.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
+dependencies = [
+ "web-time",
+ "zeroize",
+]
+
+[[package]]
+name = "rustls-platform-verifier"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
+dependencies = [
+ "core-foundation",
+ "core-foundation-sys",
+ "jni",
+ "log",
+ "once_cell",
+ "rustls",
+ "rustls-native-certs",
+ "rustls-platform-verifier-android",
+ "rustls-webpki",
+ "security-framework",
+ "security-framework-sys",
+ "webpki-root-certs",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "rustls-platform-verifier-android"
+version = "0.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
+
+[[package]]
+name = "rustls-webpki"
+version = "0.103.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
+dependencies = [
+ "aws-lc-rs",
+ "ring",
+ "rustls-pki-types",
+ "untrusted",
+]
+
[[package]]
name = "rustversion"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
+[[package]]
+name = "ryu"
+version = "1.0.23"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
+
+[[package]]
+name = "same-file"
+version = "1.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
+dependencies = [
+ "winapi-util",
+]
+
+[[package]]
+name = "schannel"
+version = "0.1.29"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
+dependencies = [
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "security-framework"
+version = "3.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
+dependencies = [
+ "bitflags 2.13.2",
+ "core-foundation",
+ "core-foundation-sys",
+ "libc",
+ "security-framework-sys",
+]
+
+[[package]]
+name = "security-framework-sys"
+version = "2.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
+dependencies = [
+ "core-foundation-sys",
+ "libc",
+]
+
+[[package]]
+name = "semver"
+version = "1.0.28"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"
+
[[package]]
name = "serde"
version = "1.0.229"
@@ -795,12 +1631,62 @@ dependencies = [
"zmij",
]
+[[package]]
+name = "serde_urlencoded"
+version = "0.7.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd"
+dependencies = [
+ "form_urlencoded",
+ "itoa",
+ "ryu",
+ "serde",
+]
+
+[[package]]
+name = "sha1"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "digest 0.11.3",
+]
+
+[[package]]
+name = "sha2"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "digest 0.11.3",
+]
+
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
+[[package]]
+name = "simd_cesu8"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "11031e251abf8611c80f460e19dbdeb54a66db918e49c65a7065b46ac7aec520"
+dependencies = [
+ "rustc_version",
+ "simdutf8",
+]
+
+[[package]]
+name = "simdutf8"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
+
[[package]]
name = "slab"
version = "0.4.12"
@@ -813,6 +1699,22 @@ version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9be42f50aa861c555654aa3a37f52f4b1074bacf4e48fe0ef7fa584e80f1f0f"
+[[package]]
+name = "socket2"
+version = "0.6.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4"
+dependencies = [
+ "libc",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "spin"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3"
+
[[package]]
name = "stable_deref_trait"
version = "1.2.1"
@@ -825,6 +1727,12 @@ version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
+[[package]]
+name = "subtle"
+version = "2.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
+
[[package]]
name = "syn"
version = "2.0.119"
@@ -847,6 +1755,15 @@ dependencies = [
"unicode-ident",
]
+[[package]]
+name = "sync_wrapper"
+version = "1.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
+dependencies = [
+ "futures-core",
+]
+
[[package]]
name = "synstructure"
version = "0.13.2"
@@ -878,6 +1795,15 @@ dependencies = [
"syn 3.0.3",
]
+[[package]]
+name = "tiny-keccak"
+version = "2.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237"
+dependencies = [
+ "crunchy",
+]
+
[[package]]
name = "tinystr"
version = "0.8.4"
@@ -910,8 +1836,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed"
dependencies = [
"bytes",
+ "libc",
+ "mio",
"pin-project-lite",
+ "socket2",
"tokio-macros",
+ "windows-sys 0.61.2",
]
[[package]]
@@ -925,6 +1855,99 @@ dependencies = [
"syn 3.0.3",
]
+[[package]]
+name = "tokio-rustls"
+version = "0.26.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67"
+dependencies = [
+ "rustls",
+ "tokio",
+]
+
+[[package]]
+name = "tokio-util"
+version = "0.7.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52"
+dependencies = [
+ "bytes",
+ "futures-core",
+ "futures-sink",
+ "pin-project-lite",
+ "tokio",
+]
+
+[[package]]
+name = "tower"
+version = "0.5.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4"
+dependencies = [
+ "futures-core",
+ "futures-util",
+ "pin-project-lite",
+ "sync_wrapper",
+ "tokio",
+ "tower-layer",
+ "tower-service",
+]
+
+[[package]]
+name = "tower-http"
+version = "0.6.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
+dependencies = [
+ "bitflags 2.13.2",
+ "bytes",
+ "futures-util",
+ "http",
+ "http-body",
+ "pin-project-lite",
+ "tower",
+ "tower-layer",
+ "tower-service",
+ "url",
+]
+
+[[package]]
+name = "tower-layer"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e"
+
+[[package]]
+name = "tower-service"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
+
+[[package]]
+name = "tracing"
+version = "0.1.44"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
+dependencies = [
+ "pin-project-lite",
+ "tracing-core",
+]
+
+[[package]]
+name = "tracing-core"
+version = "0.1.36"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
+dependencies = [
+ "once_cell",
+]
+
+[[package]]
+name = "try-lock"
+version = "0.2.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
+
[[package]]
name = "typenum"
version = "1.20.1"
@@ -952,6 +1975,12 @@ dependencies = [
"tinyvec",
]
+[[package]]
+name = "untrusted"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
+
[[package]]
name = "url"
version = "2.5.8"
@@ -982,12 +2011,43 @@ version = "1.26.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5772d71c9be8a8a6ac2117d949c5b224c1b72241bb611d9a3012edcf8af7812"
dependencies = [
- "getrandom",
+ "getrandom 0.4.3",
"js-sys",
"serde_core",
"wasm-bindgen",
]
+[[package]]
+name = "version_check"
+version = "0.9.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
+
+[[package]]
+name = "walkdir"
+version = "2.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
+dependencies = [
+ "same-file",
+ "winapi-util",
+]
+
+[[package]]
+name = "want"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
+dependencies = [
+ "try-lock",
+]
+
+[[package]]
+name = "wasi"
+version = "0.11.1+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
+
[[package]]
name = "wasm-bindgen"
version = "0.2.127"
@@ -1001,6 +2061,16 @@ dependencies = [
"wasm-bindgen-shared",
]
+[[package]]
+name = "wasm-bindgen-futures"
+version = "0.4.77"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b7777d5cc23d0e91404e53ce2d5e8ec7acae3026b16233dba62cd3246457950"
+dependencies = [
+ "js-sys",
+ "wasm-bindgen",
+]
+
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.127"
@@ -1033,6 +2103,29 @@ dependencies = [
"unicode-ident",
]
+[[package]]
+name = "wasm-streams"
+version = "0.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb"
+dependencies = [
+ "futures-util",
+ "js-sys",
+ "wasm-bindgen",
+ "wasm-bindgen-futures",
+ "web-sys",
+]
+
+[[package]]
+name = "web-sys"
+version = "0.3.104"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c435338968042f4f59a557f690a253676d47ce13ceb55d70100e7facf6620a30"
+dependencies = [
+ "js-sys",
+ "wasm-bindgen",
+]
+
[[package]]
name = "web-time"
version = "1.1.0"
@@ -1043,6 +2136,15 @@ dependencies = [
"wasm-bindgen",
]
+[[package]]
+name = "webpki-root-certs"
+version = "1.0.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b96554aa2acc8ccdb7e1c9a58a7a68dd5d13bccc69cd124cb09406db612a1c9b"
+dependencies = [
+ "rustls-pki-types",
+]
+
[[package]]
name = "which"
version = "8.0.5"
@@ -1052,12 +2154,30 @@ dependencies = [
"libc",
]
+[[package]]
+name = "winapi-util"
+version = "0.1.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
+dependencies = [
+ "windows-sys 0.61.2",
+]
+
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
+[[package]]
+name = "windows-sys"
+version = "0.52.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
+dependencies = [
+ "windows-targets",
+]
+
[[package]]
name = "windows-sys"
version = "0.61.2"
@@ -1067,6 +2187,70 @@ dependencies = [
"windows-link",
]
+[[package]]
+name = "windows-targets"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
+dependencies = [
+ "windows_aarch64_gnullvm",
+ "windows_aarch64_msvc",
+ "windows_i686_gnu",
+ "windows_i686_gnullvm",
+ "windows_i686_msvc",
+ "windows_x86_64_gnu",
+ "windows_x86_64_gnullvm",
+ "windows_x86_64_msvc",
+]
+
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
+
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
+
+[[package]]
+name = "windows_i686_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
+
+[[package]]
+name = "windows_i686_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
+
+[[package]]
+name = "windows_i686_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
+
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
+
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
+
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.52.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
+
[[package]]
name = "writeable"
version = "0.6.4"
@@ -1143,6 +2327,12 @@ dependencies = [
"synstructure",
]
+[[package]]
+name = "zeroize"
+version = "1.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
+
[[package]]
name = "zerotrie"
version = "0.2.5"
diff --git a/Cargo.toml b/Cargo.toml
index ce7227f..e209a37 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -47,6 +47,7 @@ borsh = { version = "1.8.1", default-features = false, features = ["std"] }
clap = { version = "4.6.5", features = ["derive"] }
futures-util = "0.3"
opendal = { version = "0.58.2", default-features = false }
+tokio = { version = "1", features = ["io-util"] }
unicode-casefold = "0.2"
unicode-normalization = "0.1"
uuid = { version = "1", features = ["v4"] }
diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml
index 9b54dac..497f6e1 100644
--- a/crates/core/Cargo.toml
+++ b/crates/core/Cargo.toml
@@ -30,13 +30,18 @@ blake3.workspace = true
borsh.workspace = true
futures-util.workspace = true
opendal.workspace = true
+tokio.workspace = true
unicode-casefold.workspace = true
unicode-normalization.workspace = true
uuid.workspace = true
[dev-dependencies]
-opendal = { workspace = true, features = ["services-memory"] }
-tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
+opendal = { workspace = true, features = [
+ "http-transport-reqwest",
+ "services-memory",
+ "services-s3",
+] }
+tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
[lints]
workspace = true
diff --git a/crates/core/src/content.rs b/crates/core/src/content.rs
new file mode 100644
index 0000000..a2fc81f
--- /dev/null
+++ b/crates/core/src/content.rs
@@ -0,0 +1,197 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::BTreeMap;
+
+use futures_util::TryStreamExt as _;
+use tokio::io::{AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _};
+
+use crate::{BlobRef, ContentId, Error, File, FilePart, Fs, NodeBody, Result, Tree};
+
+const DATA_PREFIX: &str = ".yinyang/data/";
+const BUFFER_BYTES: usize = 256 * 1024;
+
+impl Fs {
+ /// Upload a complete file without publishing a namespace change.
+ ///
+ /// Returns only after the backend acknowledges the immutable object.
+ /// Failed or cancelled uploads can leave unreachable data. Empty files do
+ /// not allocate an object.
+ pub async fn write_file(&self, source: &mut (impl AsyncRead + Unpin)) -> Result<File> {
+ let mut bytes = vec![0; BUFFER_BYTES];
+ let first = source
+ .read(&mut bytes)
+ .await
+ .map_err(|error| Error::from_io("read YinYang file source", error))?;
+ if first == 0 {
+ return File::new(ContentId::new(blake3::hash(&[]).into(), 0), Vec::new());
+ }
+ let path = format!("{DATA_PREFIX}{}", uuid::Uuid::new_v4().simple());
+ let mut writer = self
+ .operator
+ .writer_with(&path)
+ .if_not_exists(true)
+ .await
+ .map_err(|error| Error::from_storage("write YinYang file", error))?;
+ let result = async {
+ let mut hasher = blake3::Hasher::new();
+ let mut length = 0_u64;
+ let mut count = first;
+ loop {
+ length = length
+ .checked_add(count as u64)
+ .ok_or_else(|| Error::invalid("write YinYang file", "file length overflows"))?;
+ hasher.update(&bytes[..count]);
+ writer
+ .write(bytes[..count].to_vec())
+ .await
+ .map_err(|error| Error::from_storage("write YinYang file", error))?;
+ count = source
+ .read(&mut bytes)
+ .await
+ .map_err(|error| Error::from_io("read YinYang file source", error))?;
+ if count == 0 {
+ break;
+ }
+ }
+ writer
+ .close()
+ .await
+ .map_err(|error| Error::from_storage("close YinYang file", error))?;
+ let content = ContentId::new(hasher.finalize().into(), length);
+ File::new(
+ content,
+ vec![FilePart::new(0..length, 0, BlobRef::new(path, content))?],
+ )
+ }
+ .await;
+ if result.is_err() {
+ let _ = writer.abort().await;
+ }
+ result
+ }
+
+ /// Copy and verify a complete file with bounded transfer buffers.
+ ///
+ /// Every referenced blob is read in full, including bytes outside a part,
+ /// because its digest covers the complete blob. The assembled logical
+ /// file is checked separately. Bytes delivered before this method returns
+ /// are provisional: on any error the caller must discard the destination.
+ /// This method does not flush, close, or sync the caller's destination.
+ pub async fn read_file(
+ &self,
+ file: &File,
+ destination: &mut (impl AsyncWrite + Unpin),
+ ) -> Result<()> {
+ let mut logical_hasher = blake3::Hasher::new();
+ for part in file.parts() {
+ let reference = part.blob();
+ let path = data_path(reference)?;
+ let reader = self
+ .operator
+ .reader_with(path)
+ .chunk(BUFFER_BYTES)
+ .await
+ .map_err(read_error)?;
+ let mut stream = reader.into_stream(..).await.map_err(read_error)?;
+ let mut hasher = blake3::Hasher::new();
+ let mut offset = 0_u64;
+ let start = part.blob_offset();
+ let end = start + (part.range().end - part.range().start);
+ while let Some(buffer) = stream.try_next().await.map_err(read_error)? {
+ for bytes in buffer {
+ let next = offset
+ .checked_add(bytes.len() as u64)
+ .filter(|next| *next <= reference.content().length())
+ .ok_or_else(|| {
+ Error::corrupt("read YinYang file", "blob exceeds its declared length")
+ })?;
+ hasher.update(&bytes);
+ let copy_start = start.max(offset);
+ let copy_end = end.min(next);
+ if copy_start < copy_end {
+ let selected =
+ &bytes[(copy_start - offset) as usize..(copy_end - offset) as usize];
+ logical_hasher.update(selected);
+ destination.write_all(selected).await.map_err(|error| {
+ Error::from_io("write YinYang file destination", error)
+ })?;
+ }
+ offset = next;
+ }
+ }
+ if ContentId::new(hasher.finalize().into(), offset) != reference.content() {
+ return Err(Error::corrupt(
+ "read YinYang file",
+ "blob does not match its reference",
+ ));
+ }
+ }
+ if logical_hasher.finalize().as_bytes() != file.content().digest() {
+ return Err(Error::corrupt(
+ "read YinYang file",
+ "logical content digest does not match",
+ ));
+ }
+ Ok(())
+ }
+
+ pub(crate) async fn verify_new_files(&self, previous: &Tree, next: &Tree) -> Result<()> {
+ let previous = previous
+ .iter()
+ .map(|(_, node)| (node.id(), node))
+ .collect::<BTreeMap<_, _>>();
+ for (_, node) in next.iter() {
+ if let NodeBody::File(file) = node.body() {
+ if previous
+ .get(&node.id())
+ .is_some_and(|old| old.body() == node.body())
+ {
+ continue;
+ }
+ self.read_file(file, &mut tokio::io::sink()).await?;
+ }
+ }
+ Ok(())
+ }
+}
+
+fn data_path(reference: &BlobRef) -> Result<&str> {
+ let path = std::str::from_utf8(reference.as_bytes())
+ .map_err(|_| Error::corrupt("read YinYang file", "invalid data reference"))?;
+ let valid = path.strip_prefix(DATA_PREFIX).is_some_and(|id| {
+ id.len() == 32
+ && id
+ .bytes()
+ .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
+ });
+ if !valid {
+ return Err(Error::corrupt(
+ "read YinYang file",
+ "invalid data reference",
+ ));
+ }
+ Ok(path)
+}
+
+fn read_error(error: opendal::Error) -> Error {
+ if error.kind() == opendal::ErrorKind::NotFound {
+ Error::corrupt("read YinYang file", "referenced data is missing")
+ } else {
+ Error::from_storage("read YinYang file", error)
+ }
+}
diff --git a/crates/core/src/data.rs b/crates/core/src/data.rs
new file mode 100644
index 0000000..c6c3395
--- /dev/null
+++ b/crates/core/src/data.rs
@@ -0,0 +1,782 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Canonical, range-verifiable content and process-local preparation evidence.
+
+use std::ops::Range;
+use std::sync::Arc;
+
+use futures_util::TryStreamExt as _;
+use opendal::Operator;
+use tokio::io::{AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _};
+
+use crate::{Error, NodeId, Result};
+
+/// Fixed logical verification unit of the experimental content profile.
+pub const BLOCK_BYTES: usize = 64 * 1024;
+const FILE_MAGIC: [u8; 8] = *b"YYFILE02";
+const NODE_MAGIC: [u8; 8] = *b"YYHASH02";
+const MAX_NODE_BYTES: u32 = 512;
+
+type RefWire = ([u8; 16], u64, u32, [u8; 32]);
+type ChildWire = ([u8; 32], RefWire);
+type FileWire = ([u8; 8], [u8; 16], u64, Option<ChildWire>);
+
+/// Identity of logical bytes in the fixed YYFILE02 content profile.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub struct ContentId {
+ length: u64,
+ digest: [u8; 32],
+}
+
+impl ContentId {
+ pub const fn length(self) -> u64 {
+ self.length
+ }
+ pub const fn digest(&self) -> &[u8; 32] {
+ &self.digest
+ }
+}
+
+/// Untrusted serializable reference, not evidence that its bytes are ready.
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub struct ContentDescriptor {
+ filesystem: NodeId,
+ length: u64,
+ root: Option<Child>,
+}
+
+impl ContentDescriptor {
+ pub fn content_id(&self) -> ContentId {
+ let mut hash = domain(b"file");
+ hash.update(&self.length.to_le_bytes());
+ hash.update(&self.root.as_ref().map_or(empty_hash(), |root| root.hash));
+ ContentId {
+ length: self.length,
+ digest: hash.finalize().into(),
+ }
+ }
+
+ pub const fn filesystem(&self) -> NodeId {
+ self.filesystem
+ }
+
+ pub fn to_bytes(&self) -> Vec<u8> {
+ encode(&(
+ FILE_MAGIC,
+ *self.filesystem.as_bytes(),
+ self.length,
+ self.root.as_ref().map(Child::wire),
+ ))
+ .expect("content descriptor contains only bounded values")
+ }
+
+ pub fn from_bytes(bytes: &[u8]) -> Result<Self> {
+ if bytes.len() > MAX_NODE_BYTES as usize {
+ return Err(corrupt("descriptor is too large"));
+ }
+ let (magic, filesystem, length, root): FileWire = decode(bytes)?;
+ if magic != FILE_MAGIC {
+ return Err(Error::unsupported(
+ "decode content",
+ "unknown content profile",
+ ));
+ }
+ if (length == 0) != root.is_none() {
+ return Err(corrupt("empty content has an invalid root"));
+ }
+ let root = root.map(Child::from_wire).transpose()?;
+ Ok(Self {
+ filesystem: NodeId::from_bytes(filesystem),
+ length,
+ root,
+ })
+ }
+}
+
+/// Evidence issued by one trusted data binding after immutable preparation.
+///
+/// The handle has no public constructor or deserializer. Serialization of its
+/// descriptor loses the evidence; import must independently verify it again.
+#[derive(Clone, Debug)]
+pub struct PreparedContent {
+ descriptor: ContentDescriptor,
+ issuer: Arc<()>,
+}
+
+impl PreparedContent {
+ pub fn descriptor(&self) -> &ContentDescriptor {
+ &self.descriptor
+ }
+ pub fn content_id(&self) -> ContentId {
+ self.descriptor.content_id()
+ }
+}
+
+/// Immutable data binding for one filesystem and one storage operator.
+///
+/// Clones share preparation authority. A separately constructed binding cannot
+/// fabricate evidence accepted by this instance, even with the same filesystem ID.
+#[derive(Clone, Debug)]
+pub struct DataStore {
+ operator: Operator,
+ filesystem: NodeId,
+ issuer: Arc<()>,
+}
+
+impl DataStore {
+ pub fn new(operator: Operator, filesystem: NodeId) -> Result<Self> {
+ let capability = operator.info().capability();
+ if !capability.read
+ || !capability.write
+ || !capability.write_can_multi
+ || !capability.write_with_if_not_exists
+ {
+ return Err(Error::unsupported(
+ "open content binding",
+ "read, streaming immutable writes, and range reads are required",
+ ));
+ }
+ Ok(Self {
+ operator,
+ filesystem,
+ issuer: Arc::new(()),
+ })
+ }
+
+ /// Validate preparation authority without downloading data.
+ pub fn accept(&self, prepared: &PreparedContent) -> Result<ContentDescriptor> {
+ self.check(&prepared.descriptor)?;
+ if !Arc::ptr_eq(&self.issuer, &prepared.issuer) {
+ return Err(Error::invalid(
+ "accept prepared content",
+ "preparation authority does not match",
+ ));
+ }
+ Ok(prepared.descriptor.clone())
+ }
+
+ /// Prepare content with bounded buffers and a logarithmic hash frontier.
+ ///
+ /// This conservative binding verifies the acknowledged pack once during
+ /// preparation: an OpenDAL writer-close acknowledgement alone does not
+ /// establish end-to-end integrity. Commit never repeats this read.
+ pub async fn prepare(&self, source: &mut (impl AsyncRead + Unpin)) -> Result<PreparedContent> {
+ let mut pack = Pack::new(self).await?;
+ let result = self.prepare_into(source, &mut pack).await;
+ match result {
+ Ok(descriptor) => {
+ pack.finish().await?;
+ Ok(self.seal(descriptor))
+ }
+ Err(error) => {
+ pack.abort().await;
+ Err(error)
+ }
+ }
+ }
+
+ async fn prepare_into(
+ &self,
+ source: &mut (impl AsyncRead + Unpin),
+ pack: &mut Pack,
+ ) -> Result<ContentDescriptor> {
+ let mut frontier = Vec::<(u64, u64, Child)>::new();
+ let mut length = 0_u64;
+ loop {
+ let mut bytes = vec![0; BLOCK_BYTES];
+ let mut used = 0;
+ while used < bytes.len() {
+ let count = source
+ .read(&mut bytes[used..])
+ .await
+ .map_err(|error| Error::from_io("read content source", error))?;
+ if count == 0 {
+ break;
+ }
+ used += count;
+ }
+ if used == 0 {
+ break;
+ }
+ bytes.truncate(used);
+ let start = length / BLOCK_BYTES as u64;
+ length = length
+ .checked_add(used as u64)
+ .ok_or_else(|| Error::invalid("prepare content", "length overflows"))?;
+ let child = pack.leaf(start, &bytes).await?;
+ frontier.push((start, 1, child));
+ while frontier.len() >= 2
+ && frontier[frontier.len() - 1].1 == frontier[frontier.len() - 2].1
+ {
+ let (_, right_count, right) = frontier.pop().expect("right frontier");
+ let (start, left_count, left) = frontier.pop().expect("left frontier");
+ let count = left_count + right_count;
+ let child = pack.branch(start, count, left, right).await?;
+ frontier.push((start, count, child));
+ }
+ }
+ let root = if let Some((mut start, mut count, mut child)) = frontier.pop() {
+ while let Some((left_start, left_count, left)) = frontier.pop() {
+ start = left_start;
+ count += left_count;
+ child = pack.branch(start, count, left, child).await?;
+ }
+ debug_assert_eq!(start, 0);
+ Some(child)
+ } else {
+ None
+ };
+ Ok(ContentDescriptor {
+ filesystem: self.filesystem,
+ length,
+ root,
+ })
+ }
+
+ /// Import untrusted references by verifying every logical unit.
+ pub async fn import(&self, descriptor: &ContentDescriptor) -> Result<PreparedContent> {
+ self.read_range(descriptor, 0..descriptor.length, &mut tokio::io::sink())
+ .await?;
+ Ok(self.seal(descriptor.clone()))
+ }
+
+ /// Read only the intersecting units and their proof paths.
+ ///
+ /// Each unit is authenticated before any of its bytes reach the destination.
+ /// A later failure can leave a verified prefix, never an unverified unit.
+ pub async fn read_range(
+ &self,
+ descriptor: &ContentDescriptor,
+ range: Range<u64>,
+ destination: &mut (impl AsyncWrite + Unpin),
+ ) -> Result<()> {
+ self.check(descriptor)?;
+ if range.start > range.end || range.end > descriptor.length {
+ return Err(Error::invalid(
+ "read content range",
+ "range exceeds logical length",
+ ));
+ }
+ if range.is_empty() {
+ return Ok(());
+ }
+ let root = descriptor
+ .root
+ .as_ref()
+ .ok_or_else(|| corrupt("missing nonempty root"))?;
+ self.read_subtree(
+ root,
+ 0,
+ leaves(descriptor.length),
+ descriptor.length,
+ &range,
+ destination,
+ )
+ .await
+ }
+
+ async fn read_subtree(
+ &self,
+ child: &Child,
+ start: u64,
+ count: u64,
+ length: u64,
+ range: &Range<u64>,
+ destination: &mut (impl AsyncWrite + Unpin),
+ ) -> Result<()> {
+ match self.node(child, start, count).await? {
+ HashNode::Leaf(reference) => {
+ let bytes = self.unit(&reference, child.hash, start, length).await?;
+ let offset = start * BLOCK_BYTES as u64;
+ let begin = range.start.saturating_sub(offset) as usize;
+ let end = (range.end - offset).min(bytes.len() as u64) as usize;
+ destination
+ .write_all(&bytes[begin..end])
+ .await
+ .map_err(|error| Error::from_io("write verified content", error))?;
+ }
+ HashNode::Branch(left, right) => {
+ let left_count = split(count);
+ let boundary = (start + left_count) * BLOCK_BYTES as u64;
+ if range.start < boundary {
+ Box::pin(self.read_subtree(
+ &left,
+ start,
+ left_count,
+ length,
+ range,
+ destination,
+ ))
+ .await?;
+ }
+ if range.end > boundary {
+ Box::pin(self.read_subtree(
+ &right,
+ start + left_count,
+ count - left_count,
+ length,
+ range,
+ destination,
+ ))
+ .await?;
+ }
+ }
+ }
+ Ok(())
+ }
+
+ /// Fixed-offset update of trusted content. Work follows affected units and hash paths.
+ pub async fn overwrite_prepared(
+ &self,
+ file: &PreparedContent,
+ range: Range<u64>,
+ bytes: &[u8],
+ ) -> Result<PreparedContent> {
+ let descriptor = self.accept(file)?;
+ if range.start > range.end
+ || range.end > descriptor.length
+ || range.end - range.start != bytes.len() as u64
+ {
+ return Err(Error::invalid(
+ "overwrite prepared content",
+ "invalid replacement range",
+ ));
+ }
+ if range.is_empty() {
+ return Ok(file.clone());
+ }
+ let mut pack = Pack::new(self).await?;
+ let result = self
+ .patch(
+ descriptor.root.as_ref().expect("nonempty content"),
+ 0,
+ leaves(descriptor.length),
+ descriptor.length,
+ &range,
+ bytes,
+ &mut pack,
+ )
+ .await;
+ match result {
+ Ok(root) => {
+ pack.finish().await?;
+ Ok(self.seal(ContentDescriptor {
+ root: Some(root),
+ ..descriptor
+ }))
+ }
+ Err(error) => {
+ pack.abort().await;
+ Err(error)
+ }
+ }
+ }
+
+ async fn patch(
+ &self,
+ child: &Child,
+ start: u64,
+ count: u64,
+ length: u64,
+ range: &Range<u64>,
+ replacement: &[u8],
+ pack: &mut Pack,
+ ) -> Result<Child> {
+ let offset = start * BLOCK_BYTES as u64;
+ let subtree_end = start
+ .checked_add(count)
+ .and_then(|n| n.checked_mul(BLOCK_BYTES as u64))
+ .unwrap_or(u64::MAX)
+ .min(length);
+ if range.end <= offset || range.start >= subtree_end {
+ return Ok(child.clone());
+ }
+ match self.node(child, start, count).await? {
+ HashNode::Leaf(reference) => {
+ let mut bytes = self.unit(&reference, child.hash, start, length).await?;
+ let begin = range.start.max(offset);
+ let end = range.end.min(subtree_end);
+ bytes[(begin - offset) as usize..(end - offset) as usize].copy_from_slice(
+ &replacement[(begin - range.start) as usize..(end - range.start) as usize],
+ );
+ pack.leaf(start, &bytes).await
+ }
+ HashNode::Branch(left, right) => {
+ let left_count = split(count);
+ let left = Box::pin(self.patch(
+ &left,
+ start,
+ left_count,
+ length,
+ range,
+ replacement,
+ pack,
+ ))
+ .await?;
+ let right = Box::pin(self.patch(
+ &right,
+ start + left_count,
+ count - left_count,
+ length,
+ range,
+ replacement,
+ pack,
+ ))
+ .await?;
+ pack.branch(start, count, left, right).await
+ }
+ }
+ }
+
+ async fn node(&self, child: &Child, start: u64, count: u64) -> Result<HashNode> {
+ let bytes = self.read_extent(&child.reference, MAX_NODE_BYTES).await?;
+ let node = HashNode::decode(&bytes)?;
+ match &node {
+ HashNode::Leaf(_) if count == 1 => {}
+ HashNode::Branch(left, right)
+ if count > 1 && branch_hash(start, count, left.hash, right.hash) == child.hash => {}
+ _ => return Err(corrupt("invalid hash tree shape or proof")),
+ }
+ Ok(node)
+ }
+
+ async fn unit(
+ &self,
+ reference: &PackedRef,
+ hash: [u8; 32],
+ start: u64,
+ length: u64,
+ ) -> Result<Vec<u8>> {
+ let bytes = self.read_extent(reference, BLOCK_BYTES as u32).await?;
+ let expected = (length - start * BLOCK_BYTES as u64).min(BLOCK_BYTES as u64);
+ if bytes.len() as u64 != expected || leaf_hash(start, &bytes) != hash {
+ return Err(corrupt(
+ "verification unit does not match the pinned logical root",
+ ));
+ }
+ Ok(bytes)
+ }
+
+ fn check(&self, descriptor: &ContentDescriptor) -> Result<()> {
+ if descriptor.filesystem != self.filesystem {
+ return Err(Error::invalid("use content", "filesystem does not match"));
+ }
+ Ok(())
+ }
+
+ fn seal(&self, descriptor: ContentDescriptor) -> PreparedContent {
+ PreparedContent {
+ descriptor,
+ issuer: self.issuer.clone(),
+ }
+ }
+
+ pub(crate) async fn read_extent(&self, reference: &PackedRef, maximum: u32) -> Result<Vec<u8>> {
+ let end = reference
+ .offset
+ .checked_add(reference.length as u64)
+ .ok_or_else(|| corrupt("extent overflows"))?;
+ if reference.length == 0 || reference.length > maximum {
+ return Err(corrupt("extent length exceeds its profile bound"));
+ }
+ let bytes = self
+ .operator
+ .read_with(&self.key(reference.object))
+ .range(reference.offset..end)
+ .await
+ .map_err(storage_error)?
+ .to_bytes()
+ .to_vec();
+ if bytes.len() != reference.length as usize
+ || blake3::hash(&bytes).as_bytes() != &reference.digest
+ {
+ return Err(corrupt("extent checksum or length does not match"));
+ }
+ Ok(bytes)
+ }
+
+ fn key(&self, object: [u8; 16]) -> String {
+ format!(
+ ".yinyang/v2/{}/packs/{}",
+ uuid::Uuid::from_bytes(*self.filesystem.as_bytes()).simple(),
+ uuid::Uuid::from_bytes(object).simple()
+ )
+ }
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub(crate) struct PackedRef {
+ pub(crate) object: [u8; 16],
+ pub(crate) offset: u64,
+ pub(crate) length: u32,
+ pub(crate) digest: [u8; 32],
+}
+
+impl PackedRef {
+ fn wire(&self) -> RefWire {
+ (self.object, self.offset, self.length, self.digest)
+ }
+ fn from_wire((object, offset, length, digest): RefWire) -> Result<Self> {
+ if length == 0 || offset.checked_add(length as u64).is_none() {
+ return Err(corrupt("invalid extent"));
+ }
+ Ok(Self {
+ object,
+ offset,
+ length,
+ digest,
+ })
+ }
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+struct Child {
+ hash: [u8; 32],
+ reference: PackedRef,
+}
+
+impl Child {
+ fn wire(&self) -> ChildWire {
+ (self.hash, self.reference.wire())
+ }
+ fn from_wire((hash, reference): ChildWire) -> Result<Self> {
+ Ok(Self {
+ hash,
+ reference: PackedRef::from_wire(reference)?,
+ })
+ }
+}
+
+enum HashNode {
+ Leaf(PackedRef),
+ Branch(Child, Child),
+}
+
+impl HashNode {
+ fn bytes(&self) -> Result<Vec<u8>> {
+ match self {
+ Self::Leaf(reference) => encode(&(NODE_MAGIC, 0_u8, reference.wire())),
+ Self::Branch(left, right) => encode(&(NODE_MAGIC, 1_u8, left.wire(), right.wire())),
+ }
+ }
+
+ fn decode(bytes: &[u8]) -> Result<Self> {
+ if !bytes.starts_with(&NODE_MAGIC) {
+ return Err(corrupt("invalid hash index profile"));
+ }
+ match bytes.get(8) {
+ Some(0) => Ok(Self::Leaf(PackedRef::from_wire(decode(&bytes[9..])?)?)),
+ Some(1) => {
+ let (left, right): (ChildWire, ChildWire) = decode(&bytes[9..])?;
+ Ok(Self::Branch(
+ Child::from_wire(left)?,
+ Child::from_wire(right)?,
+ ))
+ }
+ _ => Err(corrupt("invalid hash index kind")),
+ }
+ }
+}
+
+struct Pack {
+ store: DataStore,
+ object: [u8; 16],
+ writer: opendal::Writer,
+ length: u64,
+ hasher: blake3::Hasher,
+}
+
+impl Pack {
+ async fn new(store: &DataStore) -> Result<Self> {
+ let object = *uuid::Uuid::new_v4().as_bytes();
+ let writer = store
+ .operator
+ .writer_with(&store.key(object))
+ .if_not_exists(true)
+ .chunk(5 * 1024 * 1024)
+ .await
+ .map_err(|error| Error::from_storage("prepare immutable pack", error))?;
+ Ok(Self {
+ store: store.clone(),
+ object,
+ writer,
+ length: 0,
+ hasher: blake3::Hasher::new(),
+ })
+ }
+
+ async fn append(&mut self, bytes: Vec<u8>) -> Result<PackedRef> {
+ let length = u32::try_from(bytes.len()).map_err(|_| corrupt("extent is too large"))?;
+ let reference = PackedRef {
+ object: self.object,
+ offset: self.length,
+ length,
+ digest: blake3::hash(&bytes).into(),
+ };
+ self.length = self
+ .length
+ .checked_add(length as u64)
+ .ok_or_else(|| corrupt("pack length overflows"))?;
+ self.hasher.update(&bytes);
+ self.writer
+ .write(bytes)
+ .await
+ .map_err(|error| Error::from_storage("write immutable pack", error))?;
+ Ok(reference)
+ }
+
+ async fn leaf(&mut self, position: u64, bytes: &[u8]) -> Result<Child> {
+ let hash = leaf_hash(position, bytes);
+ let data = self.append(bytes.to_vec()).await?;
+ let reference = self.append(HashNode::Leaf(data).bytes()?).await?;
+ Ok(Child { hash, reference })
+ }
+
+ async fn branch(&mut self, start: u64, count: u64, left: Child, right: Child) -> Result<Child> {
+ let hash = branch_hash(start, count, left.hash, right.hash);
+ let reference = self.append(HashNode::Branch(left, right).bytes()?).await?;
+ Ok(Child { hash, reference })
+ }
+
+ async fn finish(mut self) -> Result<()> {
+ if self.length == 0 {
+ self.abort().await;
+ return Ok(());
+ }
+ if let Err(error) = self.writer.close().await {
+ self.abort().await;
+ return Err(Error::from_storage("finish immutable pack", error));
+ }
+ let reader = self
+ .store
+ .operator
+ .reader_with(&self.store.key(self.object))
+ .chunk(BLOCK_BYTES)
+ .await
+ .map_err(storage_error)?;
+ let mut stream = reader.into_stream(..).await.map_err(storage_error)?;
+ let mut length = 0_u64;
+ let mut hash = blake3::Hasher::new();
+ while let Some(buffer) = stream.try_next().await.map_err(storage_error)? {
+ for bytes in buffer {
+ length = length
+ .checked_add(bytes.len() as u64)
+ .filter(|length| *length <= self.length)
+ .ok_or_else(|| corrupt("prepared pack is longer than expected"))?;
+ hash.update(&bytes);
+ }
+ }
+ if length != self.length || hash.finalize() != self.hasher.finalize() {
+ return Err(corrupt("prepared pack failed readback verification"));
+ }
+ Ok(())
+ }
+
+ async fn abort(&mut self) {
+ let _ = self.writer.abort().await;
+ }
+}
+
+fn leaves(length: u64) -> u64 {
+ length.div_ceil(BLOCK_BYTES as u64)
+}
+fn split(count: u64) -> u64 {
+ 1_u64 << (63 - (count - 1).leading_zeros())
+}
+
+fn domain(tag: &[u8]) -> blake3::Hasher {
+ let mut hash = blake3::Hasher::new_derive_key("Apache OpenDAL YinYang content profile 2");
+ hash.update(tag);
+ hash
+}
+fn empty_hash() -> [u8; 32] {
+ domain(b"empty").finalize().into()
+}
+fn leaf_hash(position: u64, bytes: &[u8]) -> [u8; 32] {
+ let mut hash = domain(b"leaf");
+ hash.update(&position.to_le_bytes());
+ hash.update(&(bytes.len() as u64).to_le_bytes());
+ hash.update(bytes);
+ hash.finalize().into()
+}
+fn branch_hash(start: u64, count: u64, left: [u8; 32], right: [u8; 32]) -> [u8; 32] {
+ let mut hash = domain(b"branch");
+ hash.update(&start.to_le_bytes());
+ hash.update(&count.to_le_bytes());
+ hash.update(&left);
+ hash.update(&right);
+ hash.finalize().into()
+}
+fn encode(value: &impl borsh::BorshSerialize) -> Result<Vec<u8>> {
+ borsh::to_vec(value).map_err(|error| Error::invalid("encode content", error.to_string()))
+}
+fn decode<T: borsh::BorshDeserialize>(bytes: &[u8]) -> Result<T> {
+ borsh::from_slice(bytes).map_err(|error| corrupt(error.to_string()))
+}
+fn corrupt(message: impl Into<String>) -> Error {
+ Error::corrupt("verify content", message)
+}
+fn storage_error(error: opendal::Error) -> Error {
+ if matches!(
+ error.kind(),
+ opendal::ErrorKind::NotFound | opendal::ErrorKind::RangeNotSatisfied
+ ) {
+ corrupt("referenced extent is missing")
+ } else {
+ Error::from_storage("read immutable content", error)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn content_vectors() {
+ let empty = ContentDescriptor {
+ filesystem: NodeId::from_bytes([0; 16]),
+ length: 0,
+ root: None,
+ };
+ assert_eq!(
+ blake3::Hash::from(*empty.content_id().digest())
+ .to_hex()
+ .as_str(),
+ "8b6e8e7afc13ad19dbda880a74217bc132f02679c34b4338cd318b6c264c7714"
+ );
+ assert_eq!(
+ blake3::Hash::from(leaf_hash(0, b"abc")).to_hex().as_str(),
+ "47134cf0c0612369f04c52aff76036d23a91db22a5cc8f15a4589cec1f7f9cb5"
+ );
+ assert_eq!(
+ blake3::Hash::from(leaf_hash(1, b"x")).to_hex().as_str(),
+ "4569163fa8678830f8c5b4814cb162d80d2e6044c323b593fd6b8231905b6e9f"
+ );
+ assert_eq!(
+ blake3::Hash::from(branch_hash(
+ 0,
+ 2,
+ leaf_hash(0, &vec![7; BLOCK_BYTES]),
+ leaf_hash(1, b"x")
+ ))
+ .to_hex()
+ .as_str(),
+ "4a9449de9b3ed735f25145e7ec1dbf209b8b698092e29a7f476215dcd1c8ced3"
+ );
+ }
+}
diff --git a/crates/core/src/error.rs b/crates/core/src/error.rs
index fb76738..5855a67 100644
--- a/crates/core/src/error.rs
+++ b/crates/core/src/error.rs
@@ -25,6 +25,7 @@ pub enum ErrorKind {
NotFound,
Unsupported,
Storage,
+ Io,
}
/// Error returned by the YinYang Format core.
@@ -71,6 +72,10 @@ impl Error {
pub(crate) fn from_storage(operation: &'static str, error: opendal::Error) -> Self {
Self::new(ErrorKind::Storage, operation, error.to_string())
}
+
+ pub(crate) fn from_io(operation: &'static str, error: std::io::Error) -> Self {
+ Self::new(ErrorKind::Io, operation, error.to_string())
+ }
}
impl fmt::Display for Error {
diff --git a/crates/core/src/lib.rs b/crates/core/src/lib.rs
index 028753b..9b33fca 100644
--- a/crates/core/src/lib.rs
+++ b/crates/core/src/lib.rs
@@ -17,6 +17,8 @@
//! YinYang Format values, OpenDAL persistence, and publication state machine.
+mod content;
+pub mod data;
mod error;
mod filesystem;
mod identity;
diff --git a/crates/core/src/publication.rs b/crates/core/src/publication.rs
index 39799b3..f7ebb34 100644
--- a/crates/core/src/publication.rs
+++ b/crates/core/src/publication.rs
@@ -52,7 +52,7 @@ pub enum CommitOutcome {
#[derive(Debug)]
pub struct Fs {
root: NodeId,
- operator: opendal::Operator,
+ pub(crate) operator: opendal::Operator,
}
impl Fs {
@@ -137,6 +137,8 @@ impl Fs {
.version
.tree()
.validate_successor(version.tree(), self.root)?;
+ self.verify_new_files(observed.tree(), version.tree())
+ .await?;
let number = version.number();
let blob = write_version(&self.operator, &version).await?;
match replace_head(&self.operator, &observed.head, &blob).await {
diff --git a/crates/core/tests/core.rs b/crates/core/tests/core.rs
index ae5926e..81d5e81 100644
--- a/crates/core/tests/core.rs
+++ b/crates/core/tests/core.rs
@@ -38,6 +38,11 @@ struct TestState {
fail_after_head_write: bool,
stat_calls: u64,
version_write_calls: u64,
+ fail_data_close: bool,
+ data_aborts: u64,
+ data_read_bytes: u64,
+ data_reads: u64,
+ corrupt_pack_on_close: bool,
}
#[derive(Clone, Debug)]
@@ -172,7 +177,11 @@ impl Service for TestService {
fn read(&self, _: &OperationContext, path: &str, _: OpRead) -> opendal::Result<Self::Reader> {
let state = self.state.lock().unwrap();
let object = state.objects.get(path).cloned().ok_or_else(not_found)?;
- Ok(TestReader { object })
+ Ok(TestReader {
+ object,
+ state: self.state.clone(),
+ data: path.contains("/packs/"),
+ })
}
fn write(
@@ -232,6 +241,8 @@ impl Service for TestService {
#[derive(Debug)]
struct TestReader {
object: StoredObject,
+ state: Arc<Mutex<TestState>>,
+ data: bool,
}
impl oio::Read for TestReader {
@@ -245,6 +256,11 @@ impl oio::Read for TestReader {
async fn read(&self, range: BytesRange) -> opendal::Result<(RpRead, Buffer)> {
let range = range.to_content_range(self.object.bytes.len())?;
+ if self.data {
+ let mut state = self.state.lock().unwrap();
+ state.data_read_bytes += range.len() as u64;
+ state.data_reads += 1;
+ }
Ok((
RpRead::new(metadata(&self.object)),
Buffer::from(self.object.bytes[range].to_vec()),
@@ -274,6 +290,14 @@ impl oio::Write for TestWriter {
async fn close(&mut self) -> opendal::Result<Metadata> {
let mut state = self.state.lock().unwrap();
+ if (self.path.starts_with(".yinyang/data/") || self.path.contains("/packs/"))
+ && state.fail_data_close
+ {
+ return Err(opendal::Error::new(
+ opendal::ErrorKind::Unexpected,
+ "data close failed",
+ ));
+ }
let current = state.objects.get(&self.path);
if self.if_not_exists && current.is_some() {
return Err(opendal::Error::new(
@@ -293,10 +317,14 @@ impl oio::Write for TestWriter {
}
state.next_revision += 1;
- let object = StoredObject {
+ let mut object = StoredObject {
bytes: self.bytes.clone(),
etag: format!("\"{}\"", state.next_revision),
};
+ if self.path.contains("/packs/") && state.corrupt_pack_on_close && !object.bytes.is_empty()
+ {
+ object.bytes[0] ^= 1;
+ }
let metadata = metadata(&object);
state.objects.insert(self.path.clone(), object);
if self.path == ".yinyang/head" && state.fail_after_head_write {
@@ -310,11 +338,41 @@ impl oio::Write for TestWriter {
}
async fn abort(&mut self) -> opendal::Result<()> {
+ if self.path.starts_with(".yinyang/data/") || self.path.contains("/packs/") {
+ self.state.lock().unwrap().data_aborts += 1;
+ }
self.bytes.clear();
Ok(())
}
}
+#[tokio::test]
+async fn content_preparation_requires_verified_storage_acknowledgement() {
+ use yinyang_core::data::DataStore;
+ let backend = TestBackend::default();
+ let store = DataStore::new(backend.operator(), NodeId::generate()).unwrap();
+ backend.state.lock().unwrap().fail_data_close = true;
+ assert_eq!(
+ store
+ .prepare(&mut b"abc".as_slice())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Storage
+ );
+ assert_eq!(backend.state.lock().unwrap().data_aborts, 1);
+ backend.state.lock().unwrap().fail_data_close = false;
+ backend.state.lock().unwrap().corrupt_pack_on_close = true;
+ assert_eq!(
+ store
+ .prepare(&mut b"abc".as_slice())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+}
+
fn metadata(object: &StoredObject) -> Metadata {
Metadata::new(EntryMode::FILE)
.with_content_length(object.bytes.len() as u64)
@@ -342,6 +400,163 @@ fn add_directory(tree: &Tree, name: &str, id: NodeId) -> Tree {
successor
}
+#[tokio::test]
+async fn canonical_content_range_reads_and_incremental_overwrites() {
+ use yinyang_core::data::{BLOCK_BYTES, DataStore};
+ let backend = TestBackend::default();
+ let store = DataStore::new(backend.operator(), NodeId::generate()).unwrap();
+ let bytes = (0..BLOCK_BYTES * 9 + 17)
+ .map(|n| (n % 251) as u8)
+ .collect::<Vec<_>>();
+ let file = store.prepare(&mut bytes.as_slice()).await.unwrap();
+ let second = store.prepare(&mut bytes.as_slice()).await.unwrap();
+ assert_eq!(file.content_id(), second.content_id());
+ assert_ne!(file.descriptor(), second.descriptor());
+ backend.state.lock().unwrap().data_read_bytes = 0;
+ let range = (BLOCK_BYTES as u64 - 3)..(BLOCK_BYTES as u64 + 4);
+ let mut output = Vec::new();
+ store
+ .read_range(file.descriptor(), range.clone(), &mut output)
+ .await
+ .unwrap();
+ assert_eq!(output, bytes[range.start as usize..range.end as usize]);
+ assert!(backend.state.lock().unwrap().data_read_bytes < (BLOCK_BYTES * 2 + 10_000) as u64);
+ backend.state.lock().unwrap().data_read_bytes = 0;
+ let patched = store
+ .overwrite_prepared(&file, range.clone(), b"changed")
+ .await
+ .unwrap();
+ assert!(backend.state.lock().unwrap().data_read_bytes < (BLOCK_BYTES * 5) as u64);
+ let mut expected = bytes.clone();
+ expected[range.start as usize..range.end as usize].copy_from_slice(b"changed");
+ let complete = store.prepare(&mut expected.as_slice()).await.unwrap();
+ assert_eq!(patched.content_id(), complete.content_id());
+ output.clear();
+ store
+ .read_range(patched.descriptor(), 0..expected.len() as u64, &mut output)
+ .await
+ .unwrap();
+ assert_eq!(output, expected);
+ output.clear();
+ store
+ .read_range(file.descriptor(), 0..bytes.len() as u64, &mut output)
+ .await
+ .unwrap();
+ assert_eq!(output, bytes);
+}
+
+#[tokio::test]
+async fn content_corruption_is_rejected_before_releasing_its_unit() {
+ use yinyang_core::data::{BLOCK_BYTES, DataStore};
+ let backend = TestBackend::default();
+ let store = DataStore::new(backend.operator(), NodeId::generate()).unwrap();
+ let bytes = vec![7; BLOCK_BYTES * 3];
+ let file = store.prepare(&mut bytes.as_slice()).await.unwrap();
+ let key = backend
+ .state
+ .lock()
+ .unwrap()
+ .objects
+ .keys()
+ .find(|key| key.contains("/packs/"))
+ .unwrap()
+ .clone();
+ backend
+ .state
+ .lock()
+ .unwrap()
+ .objects
+ .get_mut(&key)
+ .unwrap()
+ .bytes[0] ^= 1;
+ let mut output = Vec::new();
+ assert_eq!(
+ store
+ .read_range(file.descriptor(), 0..1, &mut output)
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+ assert!(output.is_empty());
+ store
+ .read_range(
+ file.descriptor(),
+ BLOCK_BYTES as u64..BLOCK_BYTES as u64 + 1,
+ &mut output,
+ )
+ .await
+ .unwrap();
+ assert_eq!(output, vec![7]);
+ assert_eq!(
+ store.import(file.descriptor()).await.unwrap_err().kind(),
+ ErrorKind::Corrupt
+ );
+}
+
+#[tokio::test]
+async fn preparation_evidence_cannot_be_recreated_from_a_descriptor() {
+ use yinyang_core::data::{ContentDescriptor, DataStore};
+ let backend = TestBackend::default();
+ let id = NodeId::generate();
+ let store = DataStore::new(backend.operator(), id).unwrap();
+ let impostor = DataStore::new(backend.operator(), id).unwrap();
+ let file = store.prepare(&mut &b"trusted"[..]).await.unwrap();
+ assert_eq!(
+ impostor.accept(&file).unwrap_err().kind(),
+ ErrorKind::Invalid
+ );
+ let decoded = ContentDescriptor::from_bytes(&file.descriptor().to_bytes()).unwrap();
+ let imported = impostor.import(&decoded).await.unwrap();
+ assert_eq!(impostor.accept(&imported).unwrap(), decoded);
+ let before = backend.state.lock().unwrap().data_reads;
+ store.accept(&file).unwrap();
+ assert_eq!(backend.state.lock().unwrap().data_reads, before);
+ let another = DataStore::new(backend.operator(), NodeId::generate()).unwrap();
+ assert_eq!(
+ another.import(&decoded).await.unwrap_err().kind(),
+ ErrorKind::Invalid
+ );
+}
+
+#[tokio::test]
+async fn content_profile_is_independent_of_input_chunking() {
+ use yinyang_core::data::{BLOCK_BYTES, DataStore};
+ struct ShortReads<'a>(&'a [u8]);
+ impl tokio::io::AsyncRead for ShortReads<'_> {
+ fn poll_read(
+ mut self: std::pin::Pin<&mut Self>,
+ _: &mut std::task::Context<'_>,
+ output: &mut tokio::io::ReadBuf<'_>,
+ ) -> std::task::Poll<std::io::Result<()>> {
+ let count = self.0.len().min(output.remaining()).min(17);
+ output.put_slice(&self.0[..count]);
+ self.0 = &self.0[count..];
+ std::task::Poll::Ready(Ok(()))
+ }
+ }
+ let store = DataStore::new(TestBackend::default().operator(), NodeId::generate()).unwrap();
+ for length in [
+ 0,
+ 1,
+ BLOCK_BYTES - 1,
+ BLOCK_BYTES,
+ BLOCK_BYTES + 1,
+ 5 * BLOCK_BYTES + 9,
+ ] {
+ let bytes = vec![42; length];
+ let first = store.prepare(&mut bytes.as_slice()).await.unwrap();
+ let second = store.prepare(&mut ShortReads(&bytes)).await.unwrap();
+ assert_eq!(first.content_id(), second.content_id());
+ let mut read = Vec::new();
+ store
+ .read_range(first.descriptor(), 0..length as u64, &mut read)
+ .await
+ .unwrap();
+ assert_eq!(bytes, read);
+ }
+}
+
fn advance_root_membership(tree: &mut Tree) {
let root_path = Path::root();
let root = tree
@@ -362,6 +577,319 @@ fn advance_root_membership(tree: &mut Tree) {
);
}
+fn add_file(tree: &Tree, name: &str, file: File) -> Tree {
+ let mut tree = tree.clone();
+ advance_root_membership(&mut tree);
+ tree.insert(
+ Path::new(name).unwrap(),
+ Node::file(NodeId::generate(), Generation::FIRST, false, file),
+ );
+ tree
+}
+
+#[tokio::test]
+async fn persists_file_bytes_across_reopen_and_old_versions() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let bytes = (0..900_000).map(|i| (i % 251) as u8).collect::<Vec<_>>();
+ let file = fs.write_file(&mut bytes.as_slice()).await.unwrap();
+ let initial = fs.observe().await.unwrap();
+ let id = CommitId::generate();
+ let tree = add_file(initial.tree(), "large", file.clone());
+ backend.fail_next_head_write_after_success();
+ assert_eq!(
+ fs.commit(&initial, id, tree.clone()).await.unwrap(),
+ CommitOutcome::Committed { version: 1 }
+ );
+ assert_eq!(
+ fs.commit(&initial, id, tree).await.unwrap(),
+ CommitOutcome::Committed { version: 1 }
+ );
+ let reopened = Fs::open(backend.operator()).await.unwrap();
+ let observed = reopened.observe().await.unwrap();
+ let NodeBody::File(persisted) = observed
+ .tree()
+ .get(&Path::new("large").unwrap())
+ .unwrap()
+ .body()
+ else {
+ panic!()
+ };
+ let mut output = Vec::new();
+ reopened.read_file(persisted, &mut output).await.unwrap();
+ assert_eq!(output, bytes);
+ let replacement = fs.write_file(&mut &b"new"[..]).await.unwrap();
+ let old_node = observed.tree().get(&Path::new("large").unwrap()).unwrap();
+ let mut next = observed.tree().clone();
+ next.insert(
+ Path::new("large").unwrap(),
+ Node::file(
+ old_node.id(),
+ old_node.generation().next().unwrap(),
+ false,
+ replacement,
+ ),
+ );
+ fs.commit(&observed, CommitId::generate(), next)
+ .await
+ .unwrap();
+ output.clear();
+ reopened.read_file(&file, &mut output).await.unwrap();
+ assert_eq!(output, bytes);
+}
+
+#[tokio::test]
+async fn reads_parts_offsets_and_checks_the_logical_digest() {
+ let fs = Fs::create(TestBackend::default().operator()).await.unwrap();
+ let source = fs.write_file(&mut &b"0123456789"[..]).await.unwrap();
+ let blob = source.parts()[0].blob().clone();
+ let parts = vec![
+ FilePart::new(0..3, 6, blob.clone()).unwrap(),
+ FilePart::new(3..5, 1, blob).unwrap(),
+ ];
+ let file = File::new(
+ ContentId::new(blake3::hash(b"67812").into(), 5),
+ parts.clone(),
+ )
+ .unwrap();
+ let mut output = Vec::new();
+ fs.read_file(&file, &mut output).await.unwrap();
+ assert_eq!(output, b"67812");
+ let wrong = File::new(ContentId::new([0; 32], 5), parts).unwrap();
+ assert_eq!(
+ fs.read_file(&wrong, &mut tokio::io::sink())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+}
+
+#[tokio::test]
+async fn empty_files_need_no_data_objects() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let empty = fs.write_file(&mut &b""[..]).await.unwrap();
+ assert!(empty.parts().is_empty());
+ fs.read_file(&empty, &mut tokio::io::sink()).await.unwrap();
+ assert!(
+ !backend
+ .state
+ .lock()
+ .unwrap()
+ .objects
+ .keys()
+ .any(|path| path.starts_with(".yinyang/data/"))
+ );
+}
+
+#[tokio::test]
+async fn missing_or_corrupt_data_cannot_be_published() {
+ for corruption in ["missing", "digest", "short", "long"] {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let file = fs.write_file(&mut &b"content"[..]).await.unwrap();
+ let path = std::str::from_utf8(file.parts()[0].blob().as_bytes()).unwrap();
+ {
+ let mut state = backend.state.lock().unwrap();
+ if corruption == "missing" {
+ state.objects.remove(path);
+ } else {
+ let bytes = &mut state.objects.get_mut(path).unwrap().bytes;
+ match corruption {
+ "digest" => bytes[0] ^= 1,
+ "short" => {
+ bytes.pop();
+ }
+ _ => bytes.push(0),
+ }
+ }
+ }
+ let observed = fs.observe().await.unwrap();
+ let error = fs
+ .commit(
+ &observed,
+ CommitId::generate(),
+ add_file(observed.tree(), "bad", file),
+ )
+ .await
+ .unwrap_err();
+ assert_eq!(error.kind(), ErrorKind::Corrupt, "{corruption}: {error}");
+ assert_eq!(fs.observe().await.unwrap(), observed);
+ }
+}
+
+#[tokio::test]
+async fn failed_upload_aborts_and_never_changes_head() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let observed = fs.observe().await.unwrap();
+ backend.state.lock().unwrap().fail_data_close = true;
+ assert_eq!(
+ fs.write_file(&mut &b"content"[..])
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Storage
+ );
+ assert_eq!(backend.state.lock().unwrap().data_aborts, 1);
+ assert_eq!(fs.observe().await.unwrap(), observed);
+}
+
+#[tokio::test]
+async fn invalid_data_locations_are_rejected() {
+ let fs = Fs::create(TestBackend::default().operator()).await.unwrap();
+ for path in [".yinyang/head", ".yinyang/data/../head", "external"] {
+ let content = ContentId::new(blake3::hash(b"x").into(), 1);
+ let file = File::new(
+ content,
+ vec![FilePart::new(0..1, 0, BlobRef::new(path, content)).unwrap()],
+ )
+ .unwrap();
+ assert_eq!(
+ fs.read_file(&file, &mut tokio::io::sink())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+ }
+}
+
+#[tokio::test]
+async fn verifies_unselected_blob_bytes_and_reports_destination_errors() {
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ let uploaded = fs.write_file(&mut &b"abcdef"[..]).await.unwrap();
+ let blob = uploaded.parts()[0].blob().clone();
+ let file = File::new(
+ ContentId::new(blake3::hash(b"bc").into(), 2),
+ vec![FilePart::new(0..2, 1, blob.clone()).unwrap()],
+ )
+ .unwrap();
+ let (mut sink, peer) = tokio::io::duplex(1);
+ drop(peer);
+ assert_eq!(
+ fs.read_file(&file, &mut sink).await.unwrap_err().kind(),
+ ErrorKind::Io
+ );
+ backend
+ .state
+ .lock()
+ .unwrap()
+ .objects
+ .get_mut(std::str::from_utf8(blob.as_bytes()).unwrap())
+ .unwrap()
+ .bytes[5] ^= 1;
+ assert_eq!(
+ fs.read_file(&file, &mut tokio::io::sink())
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Corrupt
+ );
+}
+
+#[tokio::test]
+async fn source_failure_after_upload_starts_aborts_the_writer() {
+ struct FailingSource(bool);
+ impl tokio::io::AsyncRead for FailingSource {
+ fn poll_read(
+ mut self: std::pin::Pin<&mut Self>,
+ _: &mut std::task::Context<'_>,
+ buffer: &mut tokio::io::ReadBuf<'_>,
+ ) -> std::task::Poll<std::io::Result<()>> {
+ if self.0 {
+ return std::task::Poll::Ready(Err(std::io::Error::other("source failed")));
+ }
+ self.0 = true;
+ buffer.put_slice(b"started");
+ std::task::Poll::Ready(Ok(()))
+ }
+ }
+ let backend = TestBackend::default();
+ let fs = Fs::create(backend.operator()).await.unwrap();
+ assert_eq!(
+ fs.write_file(&mut FailingSource(false))
+ .await
+ .unwrap_err()
+ .kind(),
+ ErrorKind::Io
+ );
+ assert_eq!(backend.state.lock().unwrap().data_aborts, 1);
+ assert_eq!(fs.observe().await.unwrap().version().number(), 0);
+}
+
+#[tokio::test]
+#[ignore = "requires an isolated S3 bucket configured with YINYANG_S3_* variables"]
+async fn s3_file_publication_and_reopen() {
+ opendal::install_default();
+ let config = std::env::vars()
+ .filter_map(|(key, value)| {
+ key.strip_prefix("YINYANG_S3_")
+ .map(|key| (key.to_ascii_lowercase(), value))
+ })
+ .collect::<Vec<_>>();
+ let operator = Operator::via_iter("s3", config).unwrap();
+ let content = yinyang_core::data::DataStore::new(operator.clone(), NodeId::generate()).unwrap();
+ let original = vec![37; 12 * 1024 * 1024 + 17];
+ let prepared = content.prepare(&mut original.as_slice()).await.unwrap();
+ let changed = content
+ .overwrite_prepared(&prepared, 65534..65538, b"test")
+ .await
+ .unwrap();
+ let mut range = Vec::new();
+ content
+ .read_range(changed.descriptor(), 65533..65539, &mut range)
+ .await
+ .unwrap();
+ assert_eq!(range, b"%test%");
+ let mut entire = Vec::new();
+ content
+ .read_range(prepared.descriptor(), 0..original.len() as u64, &mut entire)
+ .await
+ .unwrap();
+ assert_eq!(entire, original);
+ let fs = Fs::create(operator.clone()).await.unwrap();
+ let bytes = vec![37; 12 * 1024 * 1024 + 17];
+ let file = fs.write_file(&mut bytes.as_slice()).await.unwrap();
+ let observed = fs.observe().await.unwrap();
+ let name = format!("file-{}", uuid::Uuid::new_v4().simple());
+ let tree = add_file(observed.tree(), &name, file);
+ let id = CommitId::generate();
+ assert!(matches!(
+ fs.commit(&observed, id, tree.clone()).await.unwrap(),
+ CommitOutcome::Committed { .. }
+ ));
+ assert!(matches!(
+ fs.commit(&observed, id, tree).await.unwrap(),
+ CommitOutcome::Committed { .. }
+ ));
+ assert!(matches!(
+ fs.commit(
+ &observed,
+ CommitId::generate(),
+ add_directory(observed.tree(), "loser", NodeId::generate())
+ )
+ .await
+ .unwrap(),
+ CommitOutcome::Conflict { .. }
+ ));
+ let reopened = Fs::open(operator).await.unwrap();
+ let current = reopened.observe().await.unwrap();
+ let NodeBody::File(file) = current
+ .tree()
+ .get(&Path::new(name).unwrap())
+ .unwrap()
+ .body()
+ else {
+ panic!()
+ };
+ let mut output = Vec::new();
+ reopened.read_file(file, &mut output).await.unwrap();
+ assert_eq!(output, bytes);
+}
+
#[tokio::test]
async fn creates_and_reopens_one_filesystem() {
let backend = TestBackend::default();
@@ -647,7 +1175,7 @@ async fn requires_node_generation_for_file_changes() {
node,
Generation::FIRST,
false,
- File::new(ContentId::new([1; 32], 0), Vec::new()).unwrap(),
+ filesystem.write_file(&mut &b"original"[..]).await.unwrap(),
),
);
filesystem
diff --git a/specs/README.md b/specs/README.md
index 333c7a6..48f614b 100644
--- a/specs/README.md
+++ b/specs/README.md
@@ -5,6 +5,7 @@ This directory contains the maintained contracts for Apache OpenDAL™ YinYang.
| Specification | Status |
| --- | --- |
| [YinYang Format core](yinyang-format.md) | Current implementation: materialized versions and OpenDAL head publication. |
+| [Authenticated content profile](content-profile.md) | Experimental range verification and trusted preparation; independent of publication authority. |
| [Transactional filesystem](transactional-filesystem.md) | Target contract: shared filesystem and content semantics with object-storage and metadata-service publication. Not implemented; persistent encoding decisions remain open. |
A specification describes the currently supported behavior, APIs, wire formats,
diff --git a/specs/content-profile.md b/specs/content-profile.md
new file mode 100644
index 0000000..01683df
--- /dev/null
+++ b/specs/content-profile.md
@@ -0,0 +1,90 @@
+# Experimental authenticated content profile
+
+Status: implemented by `yinyang_core::data`. This profile supplies the content
+and preparation boundary for the transactional filesystem. It is not a claim
+that either publication authority is implemented by this module, or that the
+profile has been frozen for an interoperable release.
+
+## Canonical identity
+
+The maximum verification unit is 65,536 logical bytes. Split input at fixed
+multiples of this size, independent of upload chunk boundaries. The last unit
+may be partial. Every hash below uses a fresh BLAKE3 derive-key hasher with the
+context `Apache OpenDAL YinYang content profile 2`. Integers in hash inputs are
+unsigned 64-bit little-endian values. Tags are the exact ASCII bytes shown,
+without a terminator or length prefix.
+
+- Empty subtree: hash `empty`.
+- Leaf at zero-based block position P: hash `leaf || P || length || bytes`.
+- Internal subtree beginning at P with K leaves: split at the largest power of
+ two strictly smaller than K; hash `branch || P || K || left_hash || right_hash`.
+- File identity: hash `file || logical_length || subtree_hash`. Empty files use
+ the empty-subtree hash and have no extents or index root.
+
+`ContentId` is the profile identifier, logical length, and file hash. The profile
+identifier is implicit in the `YYFILE02` descriptor. Packing and physical index
+bytes never enter these logical hashes. Fixed-offset replacement keeps length
+and block boundaries unchanged and only rebuilds affected leaves and hash paths.
+
+Conformance vectors (lowercase hexadecimal):
+
+| Input | Hash |
+| --- | --- |
+| Empty file identity | `8b6e8e7afc13ad19dbda880a74217bc132f02679c34b4338cd318b6c264c7714` |
+| Leaf 0, bytes `abc` | `47134cf0c0612369f04c52aff76036d23a91db22a5cc8f15a4589cec1f7f9cb5` |
+| Leaf 1, bytes `x` | `4569163fa8678830f8c5b4814cb162d80d2e6044c323b593fd6b8231905b6e9f` |
+| Branch at 0 with two leaves: 65,536 bytes of value 7, then `x` | `4a9449de9b3ed735f25145e7ec1dbf209b8b698092e29a7f476215dcd1c8ced3` |
+
+## Physical references and proof encoding
+
+All structures use Borsh field and tuple encoding, without derive macros.
+`PackedRef` is `(object_uuid: [u8;16], offset: u64, length: u32,
+blake3_digest: [u8;32])`. The digest is ordinary BLAKE3 of exactly the referenced
+encoded bytes. An object key is derived only as
+`.yinyang/v2/<filesystem-uuid>/packs/<object-uuid>` using lowercase compact UUIDs.
+There are no caller-controlled path strings. Offset plus length must not overflow.
+
+`Child` is `(logical_subtree_hash: [u8;32], reference: PackedRef)`.
+A descriptor is `([u8;8] = YYFILE02, filesystem_uuid: [u8;16], logical_length:
+u64, root: Option<Child>)`. Empty content has no root. Hash-index nodes are:
+
+- Leaf: `([u8;8] = YYHASH02, tag: u8 = 0, data: PackedRef)`.
+- Branch: `([u8;8] = YYHASH02, tag: u8 = 1, left: Child, right: Child)`.
+
+Descriptor and index-node encodings must fit 512 bytes. Data extents contain raw,
+uncompressed bytes and fit one verification unit. The reference authenticates
+the index node independently, and child hashes authenticate selected data all
+the way to the descriptor's logical identity. Expected tree shape is derived
+from file length and block positions, not supplied by an untrusted node.
+Trailing bytes, unknown tags, invalid lengths, and invalid proofs are rejected.
+
+Data and hash-index nodes may share a container. The writer appends data and
+nodes while keeping only a logarithmic frontier and bounded transfer buffers.
+Partial reads fetch only relevant index nodes and complete intersecting units.
+Each unit's physical and logical hashes are checked before releasing its
+requested bytes. A failure in a later unit may leave a verified prefix. Unread
+units are not scrubbed as a side effect of a range read.
+
+## Preparation authority
+
+`ContentDescriptor` is serializable and untrusted. `PreparedContent` is not
+constructible or deserializable by callers: it binds the descriptor to a
+process-local `DataStore` authority shared by clones. A separately created data
+binding cannot produce handles accepted by that authority, even if given the
+same filesystem UUID. `import` verifies all units of an untrusted descriptor
+before issuing evidence. Metadata authorities must only recover published
+evidence from authenticated records in their own publication lineage.
+
+The conservative OpenDAL binding streams and hashes each newly written pack,
+closes it, and verifies the acknowledged stored bytes once during preparation.
+An incomplete, ambiguous, missing, or corrupt upload returns no handle. Once
+issued, `accept` checks authority without any storage IO. Fixed-offset updates
+use a prepared handle and read only affected old units; shared subtrees retain
+their previous readiness evidence. A serialized descriptor must first be
+imported, not promoted through a no-op update.
+
+The storage backend must provide durable immutable writes and subsequent read
+visibility. External mutation/deletion is outside the contract. This in-process
+evidence is not an authentication protocol for an untrusted remote caller.
+Compression, encryption, retention expiry, and garbage collection are not
+implemented by this profile.
diff --git a/specs/yinyang-format.md b/specs/yinyang-format.md
index 488bb20..32be92c 100644
--- a/specs/yinyang-format.md
+++ b/specs/yinyang-format.md
@@ -75,6 +75,7 @@ YinYang owns these paths below the operator root:
```text
.yinyang/head
.yinyang/versions/<object-id>
+.yinyang/data/<object-id>
```
`<object-id>` is a generated UUID encoded as 32 lowercase hexadecimal
@@ -89,6 +90,34 @@ condition mismatch is a publication conflict. Missing, truncated,
length-mismatched, or unverifiable referenced objects are corrupt data. Direct
external writes under `.yinyang/` are outside the format contract.
+## File content IO
+
+`Fs::write_file` consumes an asynchronous byte source and returns a `File`
+only after the backend acknowledges the immutable data object. It writes raw
+bytes at a fresh UUID key while computing BLAKE3 and length with bounded
+buffers. A non-empty file has one part spanning that object; an empty file
+has the BLAKE3 digest of empty bytes and no parts or object. A failed upload
+returns no file and attempts to abort the writer. Cancellation or an ambiguous
+close failure may leave unreachable data, never a published namespace entry.
+
+`Fs::read_file` streams into a caller-owned destination. Each part reads and
+verifies its entire source blob, including bytes outside the selected range;
+the assembled logical file digest is also verified. Repeated references may
+read the same blob again. Memory use does not grow with file size. Missing
+data, invalid data keys, length mismatches, and digest mismatches are `Corrupt`;
+other backend failures are `Storage`, and source/destination IO failures are
+`Io`. Bytes delivered before successful completion are provisional. Callers
+must discard them on failure and own destination flushing and durability.
+
+Before publishing, `Fs::commit` reads and verifies each new or changed file
+representation. Unchanged file bodies under the same node identity need no
+new read. This includes files constructed by callers rather than uploaded
+through `write_file`, and prevents publication of incomplete references. The
+initial implementation trades an extra read of new data for this guarantee.
+The guarantee assumes immutable data is not externally modified or deleted.
+Head conflicts can leave complete but unreachable uploads. There is no garbage
+collection yet; callers must not delete objects referenced by older versions.
+
## Persistent encoding
The payload uses [Borsh encoding](https://borsh.io/). Integers use little-endian