You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "matanper (via GitHub)" <gi...@apache.org> on 2026/10/19 09:20:24 UTC

[PR] [fix](cloud) Make storage vault path_version/shard_num work for S3 vaults [doris]

matanper opened a new pull request, #69278:
URL: https://github.com/apache/doris/pull/69278

   ### What problem does this PR solve?
   
   Issue Number: close #68790
   
   Related PR: None
   
   Problem Summary:
   
   Storage vaults support a sharded object key layout (`path_version = 1`,
   `shard_num = N`, keys `data/{murmur64(tablet_id) % N}/{tablet_id}/{rowset_id}/{seg}.dat`),
   which spreads writes of sequentially allocated tablets across key prefixes to avoid S3
   `SlowDown`/503 throttling under heavy ingest. It could not be used for S3 vaults:
   
   1. FE: `CREATE STORAGE VAULT ... ("path_version" = "1", "shard_num" = "1024")` always failed
      with `java.lang.UnsupportedOperationException`, because `CreateStorageVaultCommand#validate`
      called `remove()` on a Guava `ImmutableMap`. Invalid values (e.g. `shard_num = 0`, which
      divides by zero in BE) were not rejected.
   2. Meta-service: the `ADD_S3_VAULT` branch built a new `StorageVaultPB` with only id, name and
      obj_info, so `path_format` was silently dropped (HDFS vaults kept it).
   3. BE: `CloudMetaMgr::get_storage_vault_info` built S3 vaults with an empty `PathFormat`, so
      they always used version 0.
   4. Even with the format propagated, version 1 was unsafe:
      - File cache keys were the file name only; under version 1 every rowset's first segment is
        `0.dat`, so different rowsets/tablets would share one cache entry and return wrong data.
      - Recycler and checker hard-coded the version 0 layout, so version 1 objects would never be
        deleted after DROP/TRUNCATE/compaction, and the checker would report false data loss.
      - The table size correctness check rebuilt version 0 paths.
   
   Fix:
   
   - FE: work on a mutable copy of the properties; validate `path_version` is 0 or 1, `shard_num`
     is required and positive for version 1 and is rejected for version 0, and report non integer
     values as `AnalysisException`.
   - Meta-service: validate `path_format` (version 0/1, positive shard_num for version 1) for both
     S3 and HDFS vaults, and persist it for S3 vaults.
   - BE: pass `vault.path_format()` for S3 vaults (legacy obj infos stay on version 0); reject a
     non positive shard_num in `StorageResource`; use the rowset's own `StorageResource` in the
     table size check.
   - BE file cache: add `BlockFileCache::doris_table_cache_file_name`, which maps a version 1 path
     `.../{rowset_id}/{file}` to the version 0 file name `{rowset_id}_{file}` and leaves version 0
     names unchanged. Use it for readers, upload cache allocation, packed files, peer cache,
     segment corruption retry and compaction GC. Because the result equals the version 0 name,
     `Segment::file_cache_key`, index file names, eviction and statistics keep matching, and
     version 0 cache keys are unchanged.
   - Recycler/checker: port `murmur_hash64A` (identical to BE `HashUtil::murmur_hash64A`, same
     seed), make the path helpers take the vault `PathFormat` (default = version 0, unchanged
     output), keep each vault's format next to its accessor, and use it for rowset/tablet deletion
     (version 1 rowset prefix deletion also removes the delete bitmap, which lives outside the
     rowset directory), forward check and inverted check. Vaults with an unsupported format get no
     accessor rather than being recycled with a wrong layout.
   
   Existing vaults and vaults without `path_format` keep using version 0; nothing changes for them.
   
   ### Release note
   
   S3 storage vaults can now be created with `path_version` = 1 and `shard_num` to use the sharded
   object key layout. Invalid `path_version`/`shard_num` values are rejected at CREATE STORAGE VAULT.
   
   ### Check List (For Author)
   
   - Test: Unit Test / Regression test (added, not run locally)
       - FE UT: CreateStorageVaultCommandTest#testValidatePathFormat
       - Meta-service UT: MetaServiceTest.CreateS3VaultWithPathFormat
       - BE UT: StorageResourceTest.FileCacheName, invalid shard num death test
       - Recycler UT: RecyclerTest.vault_path_format, RecyclerTest.delete_rowset_data_path_version_1,
         CheckerTest.path_version_1, CheckerTest.inverted_check_path_version_1
       - Regression: vault_p0/create/test_create_s3_vault_path_version (.out to be generated)
       - None of these were compiled or run locally (no JDK/Maven/thirdparty toolchain available);
         C++ files formatted with clang-format 16 and build-support/check-build-hygiene.sh passes.
   - Behavior changed: Yes. CREATE STORAGE VAULT with path_version/shard_num now succeeds (it
     previously always failed) and rejects invalid combinations; S3 vaults honor path_version 1.
     Version 0 behavior and cache keys are unchanged.
   - Does this need documentation: Yes, path_version/shard_num properties of CREATE STORAGE VAULT.
   
   
   ### Notes for reviewers
   
   - The regression suite `vault_p0/create/test_create_s3_vault_path_version` needs its `.out` file generated on a cloud cluster with storage vault enabled (`-genOut`); it was not run locally.
   - Recycler/checker need the shard hash to match BE exactly; `murmur_hash64A` is ported verbatim and pinned by the shared test vector tablet 10005, shard_num 1000 -> shard 611 (same as `StorageResourceTest.RemotePath`).
   - `RefreshFSVaultVisitor` for S3 does not re-put the resource; this is fine because a vault's `path_format` cannot be altered after CREATE.
   - No new outbound requests or remote artifact loading are introduced.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) Make storage vault path_version/shard_num work for S3 vaults [doris]

Posted by "matanper (via GitHub)" <gi...@apache.org>.
matanper commented on PR #69278:
URL: https://github.com/apache/doris/pull/69278#issuecomment-6288264924

   @gavinchou thanks for taking a look!
   
   this PR doesn't introduce the layout. path_version = 1 (data/{shard}/{tablet_id}/{rowset_id}/{seg}.dat, shard = murmur_hash64A(tablet_id) % shard_num) has existed since #35953
   It's implemented in BE StorageResource, exposed as the path_version/shard_num properties of CREATE STORAGE VAULT, and works for HDFS vaults.
   
   As for why we want v1 on S3: S3 scales request throughput per key prefix (on the order of 3,500 PUT/s per partitioned prefix) and splits partitions by key range over time. Tablet ids are allocated sequentially, so after a TRUNCATE, a new table or a new partition, all new writes go to data/<adjacent tablet ids>/..., one narrow key range, until S3 re-partitions. Under heavy ingest (about 23 Gbit/s of uploads per BE) we got tens of thousands of SlowDown/503 on UploadPart, and loads failed. A hash-based first component spreads new tablets across shard_num prefixes from the first write.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) Make storage vault path_version/shard_num work for S3 vaults [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69278:
URL: https://github.com/apache/doris/pull/69278#discussion_r4356985182


##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -1909,18 +1909,18 @@ Status CloudMetaMgr::get_storage_vault_info(StorageVaultInfos* vault_infos, bool
 
     *is_vault_mode = resp.enable_storage_vault();
 
-    auto add_obj_store = [&vault_infos](const auto& obj_store) {
+    // Legacy object store infos have no path format, they always use path version 0
+    std::ranges::for_each(resp.obj_info(), [&](const auto& obj_store) {
         vault_infos->emplace_back(obj_store.id(), S3Conf::get_s3_conf(obj_store),
                                   StorageVaultPB_PathFormat {});
-    };
-
-    std::ranges::for_each(resp.obj_info(), add_obj_store);
+    });
     std::ranges::for_each(resp.storage_vault(), [&](const auto& vault) {
         if (vault.has_hdfs_info()) {
             vault_infos->emplace_back(vault.id(), vault.hdfs_info(), vault.path_format());
         }
         if (vault.has_obj_info()) {
-            add_obj_store(vault.obj_info());
+            vault_infos->emplace_back(vault.obj_info().id(), S3Conf::get_s3_conf(vault.obj_info()),
+                                      vault.path_format());

Review Comment:
   [P1] Make snapshot restore use the vault's rowset layout
   
   `RESTORE ... PROPERTIES ("storage_vault_name"="<v1 S3 vault>")` now reaches this version-1 resource, but `CloudSnapshotMgr::_create_rowset_meta` still maps each destination segment to `{new_rowset_id}_{segment_id}.dat` and derives flat index names. `CloudSnapshotLoader::download` appends those names to `remote_tablet_path()`, writing `data/{shard}/{tablet}/{rowset}_0.dat`, while the restored metadata makes normal reads request `data/{shard}/{tablet}/{rowset}/0.dat`. The copy and metadata commit can succeed and leave the restored table unreadable. Derive restore destinations, including indexes, from the selected `StorageResource` and cover RESTORE into a v1 vault.



##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -1538,6 +1570,9 @@ void MetaServiceImpl::alter_storage_vault(google::protobuf::RpcController* contr
         *instance.mutable_resource_ids()->Add() = vault.id();
         *instance.mutable_storage_vault_names()->Add() = vault.name();
         vault.mutable_obj_info()->MergeFrom(last_item);
+        if (request->vault().has_path_format()) {
+            vault.mutable_path_format()->CopyFrom(request->vault().path_format());

Review Comment:
   [P1] Isolate S3 roots before enabling a different path layout
   
   Two differently named modern S3 vaults can share the same bucket/root: `ADD_S3_VAULT` only checks duplicate object configurations in legacy `instance.obj_info()`, while new vaults are stored in separate KVs. If vault A uses v0 with tablet 10005 and vault B uses v1 with `shard_num=100000`, B's live tablet 44051 hashes to shard 10005 and stores files under `data/10005/44051/<rowset>/`. Dropping A's table makes `InstanceRecycler::recycle_tablet` recursively delete `data/10005/` through A's accessor, deleting B's committed files too. Persisting the S3 v1 format makes this cross-layout overlap reachable. Reject incompatible shared roots against existing vaults, or keep their physical namespaces disjoint, and cover recycling isolation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) Make storage vault path_version/shard_num work for S3 vaults [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69278:
URL: https://github.com/apache/doris/pull/69278#issuecomment-6263902904

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) Make storage vault path_version/shard_num work for S3 vaults [doris]

Posted by "matanper (via GitHub)" <gi...@apache.org>.
matanper commented on code in PR #69278:
URL: https://github.com/apache/doris/pull/69278#discussion_r4377070151


##########
cloud/src/meta-service/meta_service_resource.cpp:
##########
@@ -1538,6 +1570,9 @@ void MetaServiceImpl::alter_storage_vault(google::protobuf::RpcController* contr
         *instance.mutable_resource_ids()->Add() = vault.id();
         *instance.mutable_storage_vault_names()->Add() = vault.name();
         vault.mutable_obj_info()->MergeFrom(last_item);
+        if (request->vault().has_path_format()) {
+            vault.mutable_path_format()->CopyFrom(request->vault().path_format());

Review Comment:
   Addressed. `check_s3_root_path_version` (959c6ae649b, ed0bfb846bb) rejects an S3 vault whose path_version differs from any storage on the same root. It reads every storage vault KV of the instance, not only the legacy `obj_info`. It runs on both `ADD_S3_VAULT` and `add_vault_into_instance` (create_instance, ADD_BUILT_IN_VAULT, ADD_HDFS_INFO with an obj info). Same-version vaults may share a root: tablet ids are unique, and a v1 vault only deletes `data/{shard}/{tablet_id}/`. A mixed v0/v1 root is now refused at creation, so the recycling overlap you describe is not reachable.
   
   5cf5de9416c also stops comparing the endpoint, so aliases of one bucket (`https://…`, regional vs global, internal vs public) can't get around the check. Test: `MetaServiceTest.CreateS3VaultRejectsMixedPathVersionOnSameRoot` (both orders, trimmed prefixes, endpoint aliases, legacy obj_info, add_vault_into_instance).



##########
be/src/cloud/cloud_meta_mgr.cpp:
##########
@@ -1909,18 +1909,18 @@ Status CloudMetaMgr::get_storage_vault_info(StorageVaultInfos* vault_infos, bool
 
     *is_vault_mode = resp.enable_storage_vault();
 
-    auto add_obj_store = [&vault_infos](const auto& obj_store) {
+    // Legacy object store infos have no path format, they always use path version 0
+    std::ranges::for_each(resp.obj_info(), [&](const auto& obj_store) {
         vault_infos->emplace_back(obj_store.id(), S3Conf::get_s3_conf(obj_store),
                                   StorageVaultPB_PathFormat {});
-    };
-
-    std::ranges::for_each(resp.obj_info(), add_obj_store);
+    });
     std::ranges::for_each(resp.storage_vault(), [&](const auto& vault) {
         if (vault.has_hdfs_info()) {
             vault_infos->emplace_back(vault.id(), vault.hdfs_info(), vault.path_format());
         }
         if (vault.has_obj_info()) {
-            add_obj_store(vault.obj_info());
+            vault_infos->emplace_back(vault.obj_info().id(), S3Conf::get_s3_conf(vault.obj_info()),
+                                      vault.path_format());

Review Comment:
   Addressed in f0c67f89fb8. `CloudSnapshotMgr::_create_rowset_meta` maps each backup file (`{rowset}_{seg}.dat` and the V1/V2 index files) to its full target path. It builds that path with the target `StorageResource`'s layout, using the same `cpp/remote_path.h` functions BE reads with. `CloudSnapshotLoader::download` writes to that path instead of appending a flat name to `remote_tablet_path()`. Test: `CloudSnapshotMgrTest.ConvertRowsetsMapsFilesToTargetLayout` (v1 with index format V1 and V2, v0 unchanged).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) Make storage vault path_version/shard_num work for S3 vaults [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #69278:
URL: https://github.com/apache/doris/pull/69278#issuecomment-6287973512

   @matanper hi, thanks for the contribution. 
   I am wondering what problem do you want to solve by introducing
   
   data/{murmur64(tablet_id) % N}/{tablet_id}/{rowset_id}/{seg}.dat
   
   It seems does not make sense adding another part derived from tablet id in the path
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) Make storage vault path_version/shard_num work for S3 vaults [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #69278:
URL: https://github.com/apache/doris/pull/69278#issuecomment-6359194017

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org