Re: [PR] (Proposal) REST API for Table Metrics [polaris]

["dimas-b (via GitHub)" <gi...@apache.org>]

dimas-b commented on code in PR #4010:
URL: https://github.com/apache/polaris/pull/4010#discussion_r2956910468


##########
site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md:
##########
@@ -0,0 +1,720 @@
+---
+title: Table Metrics REST API
+linkTitle: Table Metrics REST API
+weight: 110
+---
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Proposal: REST API for Querying Table Metrics
+
+**Author:** Anand Sankaran
+**Date:** 2026-03-02
+**Status:** Draft Proposal
+**Target:** Apache Polaris
+
+---
+
+## Abstract
+
+This proposal defines REST API endpoints for querying table metrics (scan reports, commit reports) from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns.
+
+---
+
+## Table of Contents
+
+1. [Motivation](#1-motivation)
+2. [Use Cases](#2-use-cases)
+3. [Design Principles](#3-design-principles)
+4. [API Specification](#4-api-specification)
+5. [Authorization](#5-authorization)
+6. [OpenAPI Schema](#6-openapi-schema)
+7. [Implementation Notes](#7-implementation-notes)
+
+---
+
+## 1. Motivation
+
+Apache Polaris currently persists table metrics (scan reports, commit reports) to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics information.
+
+Adding read-only REST endpoints enables:
+- Programmatic access to metrics without database credentials
+- Integration with monitoring dashboards and alerting systems
+- Consistent authorization via Polaris RBAC
+- Pagination and filtering without writing SQL
+
+---
+
+## 2. Use Cases
+
+### 2.1 Table Health Monitoring
+- Track write patterns: files added/removed per commit, record counts, duration trends
+- Identify tables with high commit frequency or unusually large commits
+- Detect issues indicating need for compaction (many small files) or optimization
+
+### 2.2 Query Performance Analysis
+- Understand read patterns: files scanned vs skipped, planning duration
+- Identify inefficient queries with low manifest/file pruning ratios
+- Correlate performance with filter expressions and projected columns
+
+### 2.3 Capacity Planning & Chargeback
+- Aggregate metrics by table, namespace, or principal over time
+- Track storage growth trends (`total_file_size_bytes`)
+- Attribute usage to teams/users via `principal_name`
+
+### 2.4 Debugging & Troubleshooting
+- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`)
+- Investigate specific commits by `snapshot_id`
+- Trace operations via `request_id`
+
+---
+
+## 3. Design Principles
+
+| Principle | Rationale |
+|-----------|-----------|
+| **Dedicated metrics-reports namespace** | Metrics APIs use `/api/metrics-reports/v1/...` to separate from management and catalog APIs |
+| **Read-only semantics** | All endpoints are read-only; metrics are written via existing flows |
+| **Consistent pagination** | Follow `pageToken` pattern (Polaris APIs) |
+| **Flexible filtering** | Time ranges, snapshot IDs, principals - common query patterns |
+| **RBAC integration** | Leverage existing Polaris authorization model |
+| **Realm handling** | Process Polaris realms consistently with existing APIs; realm context is derived from the authenticated principal |
+| **Stable envelope** | Polaris-owned stable envelope fields for resilient client integrations; type-specific payloads are versioned independently |
+
+### 3.1 Stable Envelope
+
+To reduce coupling to any single upstream schema and keep client integrations resilient, the Metrics API SHOULD return records using a **stable envelope**: a small, Polaris-owned set of top-level fields that remain consistent across all metric report types, plus a versioned payload for the type-specific body.
+
+The envelope enables clients to reliably paginate, deduplicate, and correlate records (request IDs / trace IDs) without needing to understand the full payload schema. Type-specific details live under `payload`, identified by `payload.type` and `payload.version`.
+
+#### Envelope Fields (Conceptual)
+
+| Field | Description |
+|-------|-------------|
+| `id` | Unique identifier for the report |
+| `timestampMs` | Report timestamp (epoch milliseconds) |
+| `catalog` | Catalog identifier |
+| `realm` | Realm identifier (if applicable) |
+| `actor` | Principal/service + optional client metadata |
+| `request` | Request context (requestId, trace/span IDs) |
+| `object` | Resource identity: namespace/table identifiers and optional UUID/snapshotId |
+| `payload` | `{ type, version, data, extensions? }` |
+
+#### Compatibility / Evolution Rules
+
+1. **Envelope is additive-only**: New envelope fields may be added as optional; existing envelope fields MUST NOT change meaning or type, and MUST NOT be removed.
+
+2. **Breaking changes require versioning at the API boundary**: Any breaking envelope change requires a new major API version (e.g., `/v2/...`) rather than changing `/v1`.
+
+3. **Payload is independently versioned**: `payload.type` selects the schema family (e.g., `iceberg.metrics.scan`, `iceberg.metrics.commit`), and `payload.version` increments on breaking payload changes.
+
+4. **Unknown payloads are safe to ignore**: Clients MUST treat unknown `payload.type` or higher `payload.version` as opaque and continue to operate using envelope fields (e.g., still paginate / display metadata).
+
+5. **Payload changes within a version are additive-only**: Within a given `payload.type` + `payload.version`, new fields may be added as optional; removals/renames/type changes require a new `payload.version`.
+
+6. **Flattening is a presentation choice**: The default representation SHOULD keep domain-specific structures nested under `payload.data`; alternative "flattened" views (if needed) should be offered as an explicit, separately-versioned representation rather than redefining the canonical schema.
+
+---
+
+## 4. API Specification
+
+### 4.1 Endpoint Summary
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` | List metrics for a table (type specified via query parameter) |
+
+> **Note:** The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state.
+
+### 4.2 Path Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `catalogName` | string | Name of the catalog |
+| `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) |

Review Comment:
   Correction: the separator is configurable in IRC. The default is the same, though, AFAIK.
   
   We do not have to support configurable separators in this API because Polaris does not support that in its IRC impl., though.
   
   However, when we make actual OpenAPI spec, we probably need to review namespace encoding carefully.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org