You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2026/10/21 04:24:09 UTC

[PR] branch-4.2: [feature](inverted index) Add index_disk_usage table function #67977 [doris]

airborne12 opened a new pull request, #69365:
URL: https://github.com/apache/doris/pull/69365

   ### What problem does this PR solve?
   
   Issue Number: N/A
   
   Related PR: #67977
   
   Problem Summary:
   Backport [#67977](https://github.com/apache/doris/pull/67977), merged squash `3c6af92291b14979298a25b02b1abc264287f6f4`, to branch-4.2. The `index_disk_usage` table function reports each inverted or ANN index's physical bytes and components at tablet, rowset, or segment level. The source author, cherry-pick provenance, scan limits, replica routing, visible-version capture, query IO accounting, test assertions, and regression output are preserved.
   
   Adaptations for branch-4.2:
   
   - Override `GenericReader::get_next_block`, keep direct `init_reader`, and use `SlotDescriptor::get_data_type_ptr`. This branch has no `ReaderInitContext` adapter.
   - Traverse existing rowset segment IDs and paths, and read index-file metadata and persisted row counts from `RowsetMeta`. Missing row counts still load segment footers directly with the query IO context, without caching a transient row-count failure.
   - Use `IndexDef.IndexType` and `VariableMgr.VarAttr`. Both description slots required by the target documentation generator contain English text.
   - Select the VARIANT fixture parser using the target branch's `enable_variant_v2` convention, retaining the same JSON documents, typed path, queries, and expected output.
   - Use the existing `TabletSchema::create_block` API for the two-column integer rowset fixture. Its columns are unchanged, and all test assertions are preserved.
   - Preserve existing Hudi/Lance TVFs, scanner defaults, and session-clone tests in neighboring code. Use equivalent ranges algorithms and explicit `Status` returns to satisfy static checks.
   - Omit only the source's Storage unity skip: branch-4.2 already compiles Storage sources separately.
   
   The source PR has no linked issue; no revert or follow-up fix was found during source resolution. The new optional Thrift fields and enum value retain their source wire IDs, checked against maintained branches. No persisted index format is changed.
   
   ### Release note
   
   Add `index_disk_usage` to report the physical size and components of each index.
   
   ### Check List (For Author)
   
   Local validation on `e440788164dd07ba417d97dfae1a0719b7ccacdc`:
   
   - Standard `./build.sh --be -j48` and `./build.sh --fe -j48`: PASS. BE uses ASAN with `GLIBC_COMPATIBILITY=ON`; FE uses JDK 17. The final regression-fixture adaptation leaves all production and unit-test inputs identical to the tested e369dd033a3 candidate.
   - `./run-be-ut.sh --run --filter='IndexDiskUsage*:SniiSegmentReaderLogicalIndexesTest.*' -j48`: 55/55 tests passed in 8 suites. The same binary first listed the tests; the executed names and XML exactly match all original feature tests. No sanitizer report.
   - `./run-fe-ut.sh --run 'org.apache.doris.datasource.tvf.source.IndexDiskUsageScanNodeTest,org.apache.doris.tablefunction.IndexDiskUsageTableValuedFunctionTest,org.apache.doris.qe.SessionVariablesTest'`: 53/53 passed (14 + 23 + 16), with no failures, errors, or skips.
   - `./run-regression-test.sh --conf <isolated-cluster-config> --run -d external_table_p0/tvf -s test_index_disk_usage_tvf`: PASS against a private single-FE/BE ASAN cluster. V2, V3, SNII, component totals, all levels, position details, filters, rollup/VARIANT labels, and error cases use the original suite and expected output.
   - Full CI formatting scope on the merged snapshot with branch-4.2 `2134b43d9c0f2129823a131ebd7499d659ccf6c2`: clang-format 16 and JDK 17 `mvn clean checkstyle:check` across all 28 FE reactor modules passed. The committed PR English check passed for all 29 source files, with no exceptions.
   - Configured clang-tidy checks passed on PR lines in all 5 modified production and 5 test translation units.
   
   Static-check baseline qualification: unchanged `be/src/core/types.h:578` contains an unmatched `NOLINTEND` that stops the installed clang-tidy. An analysis-only VFS overlay removes exactly that comment; the real header and checker configuration remain unchanged, no checks are disabled, and the original/overlay preprocessed 25 MB production translation units are byte-identical. The raw baseline error, exact overlay, token-identity proof, and per-file results are retained locally.
   
   Local runtime configuration: the existing startup `SKIP_CHECK_ULIMIT=true` switch was used for this small isolated regression because the shared host's `vm.max_map_count` is 655360. The BE had 7154 mappings in the recorded post-test snapshot. No host settings or sanitizer settings were changed. These are targeted feature tests; the full BE/FE runtime suites are left to CI.
   
   - Test:
       - [x] Regression test: `external_table_p0/tvf/test_index_disk_usage_tvf`.
       - [x] Unit Test: the original feature BE/FE tests and related session-variable tests.
       - [ ] Manual test.
       - [ ] No need to test.
   - Behavior changed:
       - [x] Yes. Add the table function to branch-4.2.
   - Does this need documentation?
       - [x] Yes. Same documentation requirement as #67977; no separate website PR is linked by the source.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note.
   - [ ] Confirm test cases.
   - [ ] Confirm documentation.
   - [ ] Add branch pick label.
   
   <details>
   <summary>Source hunk audit</summary>
   
   | Source path | Source hunk | Result | Target evidence / adaptation |
   |---|---|---|---|
   | `be/src/exec/scan/meta_scanner.cpp` | `@@ -37,6 +37,7 @@` | Ported | Keep target-branch context; preserve the source change. |
   | `be/src/exec/scan/meta_scanner.cpp` | `@@ -69,6 +70,11 @@ Status MetaScanner::_open_impl(RuntimeState* state) {` | Ported | Keep target-branch context; preserve the source change. |
   | `be/src/format/table/index_disk_usage_reader.cpp` | `@@ -0,0 +1,436 @@` | Adapted | Override GenericReader::get_next_block; use SlotDescriptor::get_data_type_ptr, which returns the target branch DataTypePtr. All collection and output logic is preserved. |
   | `be/src/format/table/index_disk_usage_reader.h` | `@@ -0,0 +1,121 @@` | Adapted | Declare the branch-4.2 get_next_block virtual entry point. Remove the master-only ReaderInitContext adapter; MetaScanner still calls init_reader directly. |
   | `be/src/storage/CMakeLists.txt` | `@@ -56,6 +56,9 @@ pch_reuse(Storage)` | N-A | branch-4.2 Storage CMake ends after pch_reuse(Storage) and has no storage unity batches or STORAGE_UNITY_SKIP; all sources already compile separately. |
   | `be/src/storage/index/index_disk_usage.cpp` | `@@ -0,0 +1,581 @@` | Adapted | Use branch-4.2 contiguous segment IDs, Rowset::segment_path, RowsetMeta index-file metadata and persisted row counts; load missing row counts directly through BetaRowset::load_segment without populating the rowset row-count cache. Use equivalent ranges algorithms and explicit Status error returns to satisfy branch-4.2 static checks without changing skipped/missing-container outcomes. |
   | `be/src/storage/index/index_disk_usage.h` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `be/src/storage/index/index_file_reader.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `be/src/storage/index/index_file_reader.h` | `@@ -95,6 +95,16 @@ public:` | Ported | Keep target-branch context; preserve the source change. |
   | `be/src/storage/index/inverted/snii/reader/snii_segment_reader.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `be/src/storage/index/inverted/snii/reader/snii_segment_reader.h` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `be/src/storage/segment/page_decoder.h` | `@@ -20,6 +20,7 @@` | Ported | Keep target-branch context; preserve the source change. |
   | `be/test/format/table/index_disk_usage_reader_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `be/test/storage/index/index_disk_usage_aggregate_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `be/test/storage/index/index_disk_usage_rowset_test.cpp` | `@@ -0,0 +1,249 @@` | Adapted | Use branch-4.2 TabletSchema::create_block for the two-column integer storage fixture. This produces the same columns because the fixture has no dropped or pruned columns. Preserve all original test assertions. |
   | `be/test/storage/index/index_disk_usage_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `be/test/storage/index/inverted/snii/reader/snii_segment_reader_logical_indexes_test.cpp` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java` | `@@ -28,6 +28,7 @@ import org.apache.doris.nereids.trees.expressions.functions.table.GroupCommit;` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java` | `@@ -65,6 +66,7 @@ public class BuiltinTableValuedFunctions implements FunctionHelper {` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/main/java/org/apache/doris/datasource/tvf/source/IndexDiskUsageScanNode.java` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/IndexDiskUsage.java` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java` | `@@ -28,6 +28,7 @@ import org.apache.doris.nereids.trees.expressions.functions.table.GroupCommit;` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java` | `@@ -68,6 +69,10 @@ public interface TableValuedFunctionVisitor<R, C> {` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java` | `@@ -1154,9 +1154,8 @@ public class OlapScanNode extends ScanNode {` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java` | `@@ -95,6 +95,9 @@ public class SessionVariable implements Serializable, Writable {` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java` | `@@ -1172,6 +1175,15 @@ public class SessionVariable implements Serializable, Writable {` | Adapted | Use VariableMgr.VarAttr and both description slots required by branch-4.2 DocGenerator; keep both limits, defaults and needForward=true. Existing scanner settings remain unchanged. |
   | `fe/fe-core/src/main/java/org/apache/doris/tablefunction/IndexDiskUsageTableValuedFunction.java` | `@@ -0,0 +1,478 @@` | Adapted | Use branch-4.2 IndexDef.IndexType, which Index.getIndexType returns; preserve the source enum comparisons and all test assertions. |
   | `fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java` | `@@ -69,6 +69,8 @@ public abstract class TableValuedFunctionIf {` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/test/java/org/apache/doris/datasource/tvf/source/IndexDiskUsageScanNodeTest.java` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java` | `@@ -79,6 +79,23 @@ public class SessionVariablesTest extends TestWithFeService {` | Ported | Keep target-branch context; preserve the source change. |
   | `fe/fe-core/src/test/java/org/apache/doris/tablefunction/IndexDiskUsageTableValuedFunctionTest.java` | `@@ -0,0 +1,489 @@` | Adapted | Use branch-4.2 IndexDef.IndexType, which Index.getIndexType returns; preserve the source enum comparisons and all test assertions. |
   | `gensrc/thrift/PlanNodes.thrift` | `@@ -839,6 +839,29 @@ struct TLanceIndexMetadataParams {` | Ported | Keep target-branch context; preserve the source change. |
   | `gensrc/thrift/PlanNodes.thrift` | `@@ -860,6 +883,7 @@ struct TMetaScanRange {` | Ported | Keep target-branch context; preserve the source change. |
   | `gensrc/thrift/Types.thrift` | `@@ -671,6 +671,7 @@ enum TMetadataType {` | Ported | Keep target-branch context; preserve the source change. |
   | `regression-test/data/external_table_p0/tvf/test_index_disk_usage_tvf.out` | `All source hunks; file matches source byte-for-byte` | Ported(verbatim) |  |
   | `regression-test/suites/external_table_p0/tvf/test_index_disk_usage_tvf.groovy` | `@@ -0,0 +1,187 @@` | Adapted | Use the established branch-4.2 config-sensitive VARIANT fixture constructor: implicit JSON string conversion for legacy VARIANT and parse_to_variant for VARIANT V2. Preserve the same 1000 JSON documents, typed path, queries, assertions and expected output. |
   
   </details>
   
   <details>
   <summary>Range-diff review</summary>
   
   The source squash and backport each contain one commit. The range-diff entries were reviewed as follows:
   
   - Commit metadata: retain Jack as author and the real `cherry picked from` source SHA; add the target branch, source link, and adaptation summary.
   - Reader: `_do_get_next_block` becomes `get_next_block`; use `get_data_type_ptr`; omit the unsupported `ReaderInitContext` override because the target scanner directly calls `init_reader`.
   - Collector: replace `rowset->segments()` with existing segment IDs, paths, and rowset metadata. Keep direct footer reads, row labels, query IO context, cancellation, and error propagation. The ranges calls and explicit status branches preserve ordering and skipped-container behavior.
   - Storage CMake: omit only the unity-skip hunk because this branch has no Storage unity batches.
   - Rowset test fixture: replace two calls to `create_storage_block()` with `create_block()`. Both construct the same two integer columns; no assertion is removed or weakened.
   - Regression fixture: match the configured VARIANT representation, using implicit JSON string conversion for legacy VARIANT and `parse_to_variant` for V2. The same 1000 documents and all assertions and expected output are retained.
   - Session variables: use the target annotation type and two-description shape, preserving both defaults and forwarding.
   - FE TVF and test: import the target `IndexDef.IndexType`, with the same enum values and assertions.
   - Context entries in the meta scanner, page decoder, TVF registry/visitor, session-variable tests, and Thrift enum retain the target branch's existing code. No unrelated master imports, query-cache tests, scanner defaults, or enum reservations are copied.
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [PR] branch-4.2: [feature](inverted index) Add index_disk_usage table function #67977 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on code in PR #69365:
URL: https://github.com/apache/doris/pull/69365#discussion_r4333122272


##########
be/src/format/table/index_disk_usage_reader.cpp:
##########
@@ -0,0 +1,436 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format/table/index_disk_usage_reader.h"
+
+#include <boost/algorithm/string/case_conv.hpp>
+#include <shared_mutex>
+#include <string>
+#include <string_view>
+#include <utility>
+#include <variant>
+
+#include "cloud/cloud_tablet.h"
+#include "common/cast_set.h"
+#include "core/block/block.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "runtime/exec_env.h"
+#include "runtime/query_context.h"
+#include "runtime/runtime_state.h"
+#include "storage/rowset/rowset.h"
+#include "storage/rowset/rowset_meta.h"
+#include "storage/tablet/base_tablet.h"
+#include "storage/tablet/tablet_schema.h"
+
+namespace doris {
+
+using segment_v2::IndexDiskUsageLevel;
+using segment_v2::IndexDiskUsageRecord;
+using segment_v2::IndexDiskUsageRow;
+using segment_v2::IndexDiskUsageStructure;
+
+namespace {
+
+const std::vector<std::pair<std::string_view, IndexDiskUsageReader::Column>>& column_names() {
+    using C = IndexDiskUsageReader::Column;
+    static const std::vector<std::pair<std::string_view, C>> names = {
+            {"PARTITION_NAME", C::kPartitionName},
+            {"MATERIALIZED_INDEX_NAME", C::kMaterializedIndexName},
+            {"TABLET_ID", C::kTabletId},
+            {"BACKEND_ID", C::kBackendId},
+            {"ROWSET_ID", C::kRowsetId},
+            {"SEGMENT_ID", C::kSegmentId},
+            {"INDEX_ID", C::kIndexId},
+            {"INDEX_NAME", C::kIndexName},
+            {"INDEX_TYPE", C::kIndexType},
+            {"COLUMN_NAME", C::kColumnName},
+            {"INDEX_SUFFIX", C::kIndexSuffix},
+            {"STRUCTURE", C::kStructure},
+            {"STORAGE_FORMAT", C::kStorageFormat},
+            {"SEGMENT_COUNT", C::kSegmentCount},
+            {"ROW_COUNT", C::kRowCount},
+            {"TOTAL_BYTES", C::kTotalBytes},
+            {"DICT_BYTES", C::kDictBytes},
+            {"POSTING_BYTES", C::kPostingBytes},
+            {"POSITION_BYTES", C::kPositionBytes},
+            {"STATS_BYTES", C::kStatsBytes},
+            {"OTHER_BYTES", C::kOtherBytes},
+            {"STATS_SOURCE", C::kStatsSource},
+    };
+    return names;
+}
+
+Result<IndexDiskUsageReader::Column> column_of(const std::string& slot_name) {
+    const std::string upper = boost::to_upper_copy(slot_name);
+    for (const auto& [name, column] : column_names()) {
+        if (upper == name) {
+            return column;
+        }
+    }
+    return ResultError(Status::InternalError("unknown index_disk_usage column {}", slot_name));
+}
+
+Result<IndexDiskUsageLevel> parse_level(const std::string& level) {
+    if (level == "tablet") {
+        return IndexDiskUsageLevel::kTablet;
+    }
+    if (level == "rowset") {
+        return IndexDiskUsageLevel::kRowset;
+    }
+    if (level == "segment") {
+        return IndexDiskUsageLevel::kSegment;
+    }
+    return ResultError(Status::InvalidArgument("unsupported index_disk_usage level {}", level));
+}
+
+std::string_view structure_name(IndexDiskUsageStructure structure) {
+    switch (structure) {
+    case IndexDiskUsageStructure::kTerm:
+        return "TERM";
+    case IndexDiskUsageStructure::kBkd:
+        return "BKD";
+    case IndexDiskUsageStructure::kAnn:
+        return "ANN";
+    case IndexDiskUsageStructure::kContainer:
+        return "CONTAINER";
+    }
+    return "UNKNOWN";
+}
+
+void insert_null(IColumn* column) {
+    auto& nullable = reinterpret_cast<ColumnNullable&>(*column);
+    nullable.get_nested_column().insert_default();
+    nullable.get_null_map_data().push_back(1);
+}
+
+IColumn* non_null_nested(IColumn* column) {
+    auto& nullable = reinterpret_cast<ColumnNullable&>(*column);
+    nullable.get_null_map_data().push_back(0);
+    return nullable.get_nested_column_ptr().get();
+}
+
+void insert_int64(IColumn* column, int64_t value) {
+    assert_cast<ColumnInt64*>(non_null_nested(column))->insert_value(value);
+}
+
+void insert_int32(IColumn* column, int32_t value) {
+    assert_cast<ColumnInt32*>(non_null_nested(column))->insert_value(value);
+}
+
+void insert_string(IColumn* column, std::string_view value) {
+    assert_cast<ColumnString*>(non_null_nested(column))->insert_data(value.data(), value.size());
+}
+
+} // namespace
+
+IndexDiskUsageReader::IndexDiskUsageReader(std::vector<SlotDescriptor*> slots, RuntimeState* state,
+                                           RuntimeProfile* /*profile*/, TMetaScanRange scan_range)
+        : _state(state), _slots(std::move(slots)), _scan_range(std::move(scan_range)) {}
+
+Status IndexDiskUsageReader::init_reader() {
+    if (!_scan_range.__isset.index_disk_usage_params) {
+        return Status::InvalidArgument("index_disk_usage scan range has no parameters");
+    }
+    const TIndexDiskUsageMetadataParams& params = _scan_range.index_disk_usage_params;
+    _level = DORIS_TRY(parse_level(params.level));
+    _options.position_detail = params.position_detail;
+    _options.index_ids.insert(params.index_ids.begin(), params.index_ids.end());
+    _options.check_cancelled = [state = _state]() {
+        RETURN_IF_CANCELLED(state);
+        return Status::OK();
+    };
+    // Index file reads are accounted to the query like the reads of any other scan.
+    _io_ctx = io::IOContext {
+            .reader_type = ReaderType::READER_QUERY,
+            .query_id = &_state->query_id(),
+            .is_inverted_index = true,
+    };
+    _io_ctx.inverted_index_snii_read_no_write_file_cache =
+            _state->query_options().inverted_index_snii_read_no_write_file_cache;
+    if (auto* query_ctx = _state->get_query_ctx(); query_ctx != nullptr) {
+        _io_ctx.remote_scan_cache_write_limiter = query_ctx->remote_scan_cache_write_limiter();
+    }
+    _options.io_ctx = &_io_ctx;
+    _slot_columns.clear();
+    for (const SlotDescriptor* slot : _slots) {
+        const Column column = DORIS_TRY(column_of(slot->col_name()));
+        _slot_columns.push_back(column);
+    }
+    return Status::OK();
+}
+
+Status IndexDiskUsageReader::get_next_block(Block* block, size_t* read_rows, bool* eof) {
+    const auto& tablets = _scan_range.index_disk_usage_params.tablets;
+    *read_rows = 0;
+    while (_next_tablet < tablets.size()) {
+        RETURN_IF_CANCELLED(_state);
+        const TIndexDiskUsageTablet& target = tablets[_next_tablet++];
+        std::vector<IndexDiskUsageRow> rows;

Review Comment:
   [P2] Account the per-tablet row buffers to the query tracker
   
   `rows` retains a record for every segment/index-suffix pair of a tablet before aggregation; rowset/tablet aggregation also allocates a plain `std::map` and vector. On this branch these standard-allocator allocations bypass `MemTracker` (`jemalloc_hook.cpp` only forwards to jemalloc), so the buffers are invisible to the query's `mem_limit` until output columns are allocated. A single allowed tablet with many segments and extracted VARIANT paths can therefore exceed its query budget despite `index_disk_usage_max_tablets`, creating BE memory pressure. Use allocator-aware containers or explicit reservation/accounting for these buffers and the aggregation structures.



##########
be/src/format/table/index_disk_usage_reader.cpp:
##########
@@ -0,0 +1,436 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format/table/index_disk_usage_reader.h"
+
+#include <boost/algorithm/string/case_conv.hpp>
+#include <shared_mutex>
+#include <string>
+#include <string_view>
+#include <utility>
+#include <variant>
+
+#include "cloud/cloud_tablet.h"
+#include "common/cast_set.h"
+#include "core/block/block.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "runtime/exec_env.h"
+#include "runtime/query_context.h"
+#include "runtime/runtime_state.h"
+#include "storage/rowset/rowset.h"
+#include "storage/rowset/rowset_meta.h"
+#include "storage/tablet/base_tablet.h"
+#include "storage/tablet/tablet_schema.h"
+
+namespace doris {
+
+using segment_v2::IndexDiskUsageLevel;
+using segment_v2::IndexDiskUsageRecord;
+using segment_v2::IndexDiskUsageRow;
+using segment_v2::IndexDiskUsageStructure;
+
+namespace {
+
+const std::vector<std::pair<std::string_view, IndexDiskUsageReader::Column>>& column_names() {
+    using C = IndexDiskUsageReader::Column;
+    static const std::vector<std::pair<std::string_view, C>> names = {
+            {"PARTITION_NAME", C::kPartitionName},
+            {"MATERIALIZED_INDEX_NAME", C::kMaterializedIndexName},
+            {"TABLET_ID", C::kTabletId},
+            {"BACKEND_ID", C::kBackendId},
+            {"ROWSET_ID", C::kRowsetId},
+            {"SEGMENT_ID", C::kSegmentId},
+            {"INDEX_ID", C::kIndexId},
+            {"INDEX_NAME", C::kIndexName},
+            {"INDEX_TYPE", C::kIndexType},
+            {"COLUMN_NAME", C::kColumnName},
+            {"INDEX_SUFFIX", C::kIndexSuffix},
+            {"STRUCTURE", C::kStructure},
+            {"STORAGE_FORMAT", C::kStorageFormat},
+            {"SEGMENT_COUNT", C::kSegmentCount},
+            {"ROW_COUNT", C::kRowCount},
+            {"TOTAL_BYTES", C::kTotalBytes},
+            {"DICT_BYTES", C::kDictBytes},
+            {"POSTING_BYTES", C::kPostingBytes},
+            {"POSITION_BYTES", C::kPositionBytes},
+            {"STATS_BYTES", C::kStatsBytes},
+            {"OTHER_BYTES", C::kOtherBytes},
+            {"STATS_SOURCE", C::kStatsSource},
+    };
+    return names;
+}
+
+Result<IndexDiskUsageReader::Column> column_of(const std::string& slot_name) {
+    const std::string upper = boost::to_upper_copy(slot_name);
+    for (const auto& [name, column] : column_names()) {
+        if (upper == name) {
+            return column;
+        }
+    }
+    return ResultError(Status::InternalError("unknown index_disk_usage column {}", slot_name));
+}
+
+Result<IndexDiskUsageLevel> parse_level(const std::string& level) {
+    if (level == "tablet") {
+        return IndexDiskUsageLevel::kTablet;
+    }
+    if (level == "rowset") {
+        return IndexDiskUsageLevel::kRowset;
+    }
+    if (level == "segment") {
+        return IndexDiskUsageLevel::kSegment;
+    }
+    return ResultError(Status::InvalidArgument("unsupported index_disk_usage level {}", level));
+}
+
+std::string_view structure_name(IndexDiskUsageStructure structure) {
+    switch (structure) {
+    case IndexDiskUsageStructure::kTerm:
+        return "TERM";
+    case IndexDiskUsageStructure::kBkd:
+        return "BKD";
+    case IndexDiskUsageStructure::kAnn:
+        return "ANN";
+    case IndexDiskUsageStructure::kContainer:
+        return "CONTAINER";
+    }
+    return "UNKNOWN";
+}
+
+void insert_null(IColumn* column) {
+    auto& nullable = reinterpret_cast<ColumnNullable&>(*column);
+    nullable.get_nested_column().insert_default();
+    nullable.get_null_map_data().push_back(1);
+}
+
+IColumn* non_null_nested(IColumn* column) {
+    auto& nullable = reinterpret_cast<ColumnNullable&>(*column);
+    nullable.get_null_map_data().push_back(0);
+    return nullable.get_nested_column_ptr().get();
+}
+
+void insert_int64(IColumn* column, int64_t value) {
+    assert_cast<ColumnInt64*>(non_null_nested(column))->insert_value(value);
+}
+
+void insert_int32(IColumn* column, int32_t value) {
+    assert_cast<ColumnInt32*>(non_null_nested(column))->insert_value(value);
+}
+
+void insert_string(IColumn* column, std::string_view value) {
+    assert_cast<ColumnString*>(non_null_nested(column))->insert_data(value.data(), value.size());
+}
+
+} // namespace
+
+IndexDiskUsageReader::IndexDiskUsageReader(std::vector<SlotDescriptor*> slots, RuntimeState* state,
+                                           RuntimeProfile* /*profile*/, TMetaScanRange scan_range)
+        : _state(state), _slots(std::move(slots)), _scan_range(std::move(scan_range)) {}
+
+Status IndexDiskUsageReader::init_reader() {
+    if (!_scan_range.__isset.index_disk_usage_params) {
+        return Status::InvalidArgument("index_disk_usage scan range has no parameters");
+    }
+    const TIndexDiskUsageMetadataParams& params = _scan_range.index_disk_usage_params;
+    _level = DORIS_TRY(parse_level(params.level));
+    _options.position_detail = params.position_detail;
+    _options.index_ids.insert(params.index_ids.begin(), params.index_ids.end());
+    _options.check_cancelled = [state = _state]() {
+        RETURN_IF_CANCELLED(state);
+        return Status::OK();
+    };
+    // Index file reads are accounted to the query like the reads of any other scan.
+    _io_ctx = io::IOContext {
+            .reader_type = ReaderType::READER_QUERY,
+            .query_id = &_state->query_id(),
+            .is_inverted_index = true,
+    };
+    _io_ctx.inverted_index_snii_read_no_write_file_cache =
+            _state->query_options().inverted_index_snii_read_no_write_file_cache;
+    if (auto* query_ctx = _state->get_query_ctx(); query_ctx != nullptr) {
+        _io_ctx.remote_scan_cache_write_limiter = query_ctx->remote_scan_cache_write_limiter();
+    }
+    _options.io_ctx = &_io_ctx;
+    _slot_columns.clear();
+    for (const SlotDescriptor* slot : _slots) {
+        const Column column = DORIS_TRY(column_of(slot->col_name()));
+        _slot_columns.push_back(column);
+    }
+    return Status::OK();
+}
+
+Status IndexDiskUsageReader::get_next_block(Block* block, size_t* read_rows, bool* eof) {
+    const auto& tablets = _scan_range.index_disk_usage_params.tablets;
+    *read_rows = 0;
+    while (_next_tablet < tablets.size()) {
+        RETURN_IF_CANCELLED(_state);
+        const TIndexDiskUsageTablet& target = tablets[_next_tablet++];
+        std::vector<IndexDiskUsageRow> rows;
+        TabletSchemaSPtr current_schema;
+        RETURN_IF_ERROR(_collect_tablet(target, &rows, &current_schema));
+        rows = segment_v2::aggregate_index_disk_usage(std::move(rows), _level);
+        if (rows.empty()) {
+            continue;
+        }
+        RETURN_IF_ERROR(_fill_block(block, target, *current_schema, rows));
+        *read_rows = rows.size();
+        *eof = false;
+        return Status::OK();
+    }
+    *eof = true;
+    return Status::OK();
+}
+
+Status IndexDiskUsageReader::_collect_tablet(const TIndexDiskUsageTablet& target,
+                                             std::vector<IndexDiskUsageRow>* rows,
+                                             TabletSchemaSPtr* current_schema) const {
+    const TabletRowsets captured =
+            DORIS_TRY(capture_tablet_rowsets(target.tablet_id, target.version));
+    *current_schema = label_schema(captured.tablet->tablet_schema(), captured.rowsets);
+
+    const io::IOContext io_ctx = tablet_io_context(_io_ctx, captured.tablet->ttl_seconds());

Review Comment:
   [P2] Pass the tablet's absolute file-cache expiration
   
   `ttl_seconds()` is a duration, but `IOContext.expiration_time` expects an absolute deadline on this branch. For example, after a tablet's `creation_time + 3600` deadline has passed, this path still passes `3600`, while ordinary scans use `file_cache_ttl_expiration_time()` and pass zero. `CacheContext` treats every nonzero value as TTL, so this TVF's uncached index/footer reads re-admit expired data into the protected TTL queue until background reconciliation, potentially displacing useful query cache entries. Pass `captured.tablet->file_cache_ttl_expiration_time()` here and update the helper/test to use absolute expiration semantics, including an expired-table case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org