[I] Standardize temporal column statistics across pandas, Polars, and the UI [hamilton]

["skrawcz (via GitHub)" <gi...@apache.org>]

skrawcz opened a new issue, #1737:
URL: https://github.com/apache/hamilton/issues/1737

   ## Problem
   
   Hamilton's observability schema currently treats several distinct temporal concepts as if they were JavaScript-compatible datetimes. pandas and Polars consequently produce different payloads for equivalent columns, and some payloads either cannot be serialized or render incorrectly in the UI.
   
   This was identified while reviewing #1675 and Jernej's comment in https://github.com/apache/hamilton/pull/1675#discussion_r3795651892. PR #1675 should remain a focused fix: Polars `Time` and `Duration` are routed to the existing `unhandled` path so they no longer break datetime rendering. This issue tracks the complete cross-backend solution.
   
   ### Current behavior
   
   | Logical type | pandas | Polars | UI consequence |
   |---|---|---|---|
   | Datetime | `base_data_type=\"datetime\"`; ISO-like statistics | `base_data_type=\"datetime\"`; ISO-like statistics | Existing datetime renderer generally works |
   | Date | Python `date` object columns normally become `unhandled` | `base_data_type=\"datetime\"`; min/max are `YYYY-MM-DD` | `new Date(\"YYYY-MM-DD\")` is interpreted as UTC and can display the previous day in negative-offset timezones |
   | Time | Object columns normally become `unhandled` | `unhandled` after #1675 | No rich statistics, but safe |
   | Duration/timedelta | `base_data_type=\"datetime\"`; min/max/mean/quantiles contain raw `pd.Timedelta` | `unhandled` after #1675 | pandas payload fails ordinary JSON serialization; duration strings are not valid JavaScript Dates anyway |
   
   The frontend datetime path in `ui/frontend/src/components/dashboard/Runs/Task/result-summaries/DAGWorksDescribe.tsx` calls `new Date(...)` for range, quantile, and histogram values. That operation does not preserve Date semantics and cannot parse Time or Duration values.
   
   The generated frontend datetime type and backend Pydantic datetime model also do not declare all fields that producers currently emit, which prevents static typing from catching contract drift.
   
   ## Desired solution
   
   Define one explicit temporal statistics contract shared by pandas, Polars, the backend schema, generated frontend types, and UI renderers.
   
   ### Logical types
   
   Use distinct `base_data_type` values for:
   
   1. `datetime`
   2. `date`
   3. `time`
   4. `duration`
   
   Do not route Date, Time, or Duration through the datetime renderer.
   
   ### Canonical JSON representations
   
   The precise representation should be documented and applied identically by both producers. A suggested starting point:
   
   - **Datetime:** ISO 8601 datetime strings with an explicit, documented timezone policy. Preserve offsets for timezone-aware values.
   - **Date:** `YYYY-MM-DD` strings. Treat them as calendar dates in the UI; do not pass them directly to `new Date(string)`.
   - **Time:** ISO local-time strings such as `HH:mm:ss[.ffffff]`, with an explicit policy for timezone-aware time values if supported.
   - **Duration:** a numeric canonical unit, preferably integer microseconds, for min/max/mean/quantiles and charting. Avoid display-formatted strings as the transport representation. If ISO 8601 durations are selected instead, the UI must parse them explicitly and tests must cover ordering/charting.
   
   All emitted statistics—including min, max, mean, standard deviation, quantiles, and histogram boundaries—must be JSON-safe and use the canonical representation for their logical type.
   
   A schema-version bump may be required if adding `base_data_type` values is not compatible with the current `dagworks_describe` v0.0.3 contract.
   
   ### UI behavior
   
   Add explicit dispatch and rendering for each type:
   
   - **Datetime:** retain the existing renderer after confirming timezone behavior.
   - **Date:** render calendar dates without UTC conversion or timezone shifting. Construct from components or display the canonical value through a date-aware formatter.
   - **Time:** render wall-clock values without inventing a date or applying a browser timezone.
   - **Duration:** format canonical numeric values into readable durations and chart them numerically.
   
   Unsupported or malformed values should degrade to the existing unhandled presentation rather than producing `Invalid Date` or `NaN`.
   
   ### Schema and generated types
   
   Update the backend observability models and generated frontend API types so each temporal statistics shape declares the fields actually emitted and consumed. Avoid casting datetime records to numeric statistics types in the frontend.
   
   ## Acceptance criteria
   
   - [ ] Equivalent pandas and Polars columns produce the same logical `base_data_type` and value shape for Datetime, Date, Time, and Duration.
   - [ ] Every pandas and Polars temporal payload passes JSON serialization without custom fallback encoders.
   - [ ] Datetime timezone/offset behavior is documented and tested.
   - [ ] Date values display the same calendar day in positive and negative UTC-offset browser timezones.
   - [ ] Time values display without an invented date or timezone conversion.
   - [ ] Duration values render and sort/chart numerically with documented precision and unit.
   - [ ] The frontend never invokes the datetime renderer for Date, Time, or Duration.
   - [ ] Malformed/unsupported temporal values degrade safely instead of rendering `Invalid Date`/`NaN`.
   - [ ] Backend and generated frontend types match the actual payload fields.
   - [ ] Cross-backend contract tests cover min, max, mean, std, quantiles, histogram boundaries where applicable, missing values, and empty/all-null columns.
   - [ ] Frontend tests run under at least one negative-offset timezone (for example `America/Los_Angeles`) and verify Date behavior.
   - [ ] Upgrade/schema compatibility is documented, including how older SDK payloads are rendered.
   
   ## Suggested implementation areas
   
   - pandas routing and serialization:
     - `ui/sdk/src/hamilton_sdk/tracking/pandas_stats.py`
     - `ui/sdk/src/hamilton_sdk/tracking/pandas_col_stats.py`
   - Polars routing and serialization:
     - `ui/sdk/src/hamilton_sdk/tracking/polars_stats.py`
     - `ui/sdk/src/hamilton_sdk/tracking/polars_col_stats.py`
   - Observability schema/models and generated API types
   - UI grouping and renderers:
     - `ui/frontend/src/components/dashboard/Runs/Task/result-summaries/DAGWorksDescribe.tsx`
   - Tests:
     - `ui/sdk/tests/tracking/test_pandas_stats.py`
     - `ui/sdk/tests/tracking/test_pandas_col_stats.py`
     - `ui/sdk/tests/tracking/test_polars_stats.py`
     - `ui/sdk/tests/tracking/test_polars_col_stats.py`
     - focused frontend temporal renderer/contract tests
   
   ## Incremental path
   
   Until this contract and its renderers exist, the safe schema-v0.0.3 fallback is for only true Datetime columns to use the datetime path and for unsupported Date/Time/Duration values to use `unhandled`. PR #1675 implements that fallback for Polars Time/Duration and should not be expanded into this larger schema/UI project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@hamilton.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org