You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/08/13 08:22:30 UTC

[PR] Allow non-immutable ingestion transforms for offline tables [pinot]

xiangfu0 opened a new pull request, #19244:
URL: https://github.com/apache/pinot/pull/19244

## Summary

Allow table-level ingestion transforms on `OFFLINE` tables to use registered scalar functions from any volatility category. New or changed `REALTIME` table ingestion transforms continue to require `IMMUTABLE` functions.

The rest of transform validation remains in place, including expression parsing, destination-column validation, self-reference checks, and the Groovy policy. Schema-level transforms and `postPartialUpsertTransformConfigs` also retain their existing volatility checks.

## Motivation

An offline upsert table needs a comparison value to choose the latest row for each primary key. Daily Parquet inputs can contain only a subset of the key space and may not contain a physical timestamp or row-version column. A value that is fixed for every ingestion run cannot order updates across those daily drops.

This change lets an offline table declare a generated comparison column in the Pinot schema and populate it with `now()` or another explicitly selected non-immutable transform, without first rewriting the Parquet files. The policy change is scoped to finite, task-driven `OFFLINE` ingestion and is not applied to `REALTIME` ingestion.

## Usage

Declare the generated comparison column in the Pinot schema even though it is absent from the source Parquet file:

```json
{
"schemaName": "dailyDimension",
"dimensionFieldSpecs": [
{
"name": "id",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "ingestionTime",
"dataType": "LONG"
}
],
"primaryKeyColumns": ["id"]
}
```

Configure full upsert and populate the comparison column during offline ingestion:

```json
{
"tableName": "dailyDimension",
"tableType": "OFFLINE",
"segmentsConfig": {
"segmentPushType": "APPEND",
"replication": "1"
},
"routing": {
"instanceSelectorType": "strictReplicaGroup"
},
"tableIndexConfig": {
"segmentPartitionConfig": {
"columnPartitionMap": {
"id": {
"functionName": "Murmur",
"numPartitions": 1
}
}
}
},
"upsertConfig": {
"mode": "FULL",
"comparisonColumns": ["ingestionTime"]
},
"ingestionConfig": {
"transformConfigs": [
{
"columnName": "ingestionTime",
"transformFunction": "now()"
}
]
}
}
```

The Parquet rows must contain `id`, but they do not need to contain `ingestionTime`. Offline upsert still requires primary-key partitioning and strict-replica-group routing.

## Compatibility

- `OFFLINE` table-level ingestion transforms accept `IMMUTABLE`, `STABLE`, and `VOLATILE` functions.
- New or changed `REALTIME` table-level ingestion transforms continue to reject non-immutable functions.
- An identical non-immutable transform already stored on a `REALTIME` table remains grandfathered so unrelated config updates are not blocked.
- Schema-level and post-partial-upsert transforms continue to require immutable functions.
- Unknown functions, invalid expressions, and disabled Groovy transforms remain rejected.

## Validation

- `TableConfigUtilsTest`: 78 tests passed.
- `SchemaUtilsTest`: 15 tests passed.
- `PinotTableRestletResourceTest` and `TableConfigsRestletResourceTest`: 45 tests passed.
- Spotless, Checkstyle, license formatting/checking, and `git diff --check` passed for all affected modules.

## Draft status and known correctness gap

This PR changes validation policy only; it does not make volatile functions batch-stable. Row-based offline segment generation currently evaluates transforms once while gathering statistics and again while indexing after rewinding the input. A function such as `now()` or `rand()` can therefore produce different values between the two passes. Dictionary-encoded columns can resolve a second-pass value that was absent from the first-pass dictionary to an incorrect dictionary ID, while no-dictionary columns can store values inconsistent with first-pass min/max metadata. Either case can select the wrong offline-upsert winner.

Values can also differ across records, retries, rebuilds, and backfills, and registered functions that depend on unavailable runtime context can validate but fail during batch evaluation.

This should remain a draft discussion PR until segment generation either materializes and reuses transformed rows or provides a defined build-stable evaluation mode, with an end-to-end segment-generation/readback regression test.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org