You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/10/21 22:50:29 UTC
[PR] Add a default JSON conversion hook to ColumnReader [pinot]
xiangfu0 opened a new pull request, #19996:
URL: https://github.com/apache/pinot/pull/19996
Readers currently expose only materialized values. Add default `ColumnReader.getJsonValue(docId, jsonTransformer)` so specialized readers can avoid repeated materialization while ordinary readers keep the existing conversion behavior.
Usage: `Object json = reader.getJsonValue(docId, jsonTransformer);` Pass a deterministic JSON converter; the default calls `jsonTransformer.transform(getValue(docId))`, including nulls, and propagates failures. Overrides may reuse immutable results.
Validation: all 9 `ColumnReaderTest` cases pass; formatting, license and Checkstyle checks pass; `pinot-spi` still compiles to Java 11 bytecode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #19996: Add a default value transformation hook to ColumnReader
URL: https://github.com/apache/pinot/pull/19996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19996:
URL: https://github.com/apache/pinot/pull/19996#discussion_r4352276383
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,27 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ /// The default invokes [ColumnTransformer#transform(Object)] even when [ColumnTransformer#isNoOp()] returns `true`.
+ ///
+ /// Callers must supply a deterministic transformer: repeated transformations of the same value must produce
+ /// equivalent results. Callers must not depend on invocation counts or transformer side effects.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values, reuse immutable conversion results,
+ /// or skip no-op transformations. Overrides must preserve the conversion result, including nulls and conversion
+ /// failures. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getTransformedValue(int docId, ColumnTransformer transformer)
+ throws IOException {
+ return transformer.transform(getValue(docId));
Review Comment:
Can we filter no-op transformers in the caller during pipeline setup and assert that precondition here?
```java
assert !transformer.isNoOp() : "No-op transformers must be filtered by the caller";
return transformer.transform(getValue(docId));
```
With assertions disabled, this avoids evaluating `isNoOp()` for each document. Please document that callers must supply a non-no-op transformer and remove the statements allowing no-op inputs or overrides to skip them. An assertion-enabled test can cover rejection.
This matters for existing implementations: `SanitizationColumnTransformer` for an INT column reports `isNoOp() == true`, but calling `transform(42)` dereferences null configuration. Making these inputs invalid for this API keeps that handling in the caller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19996:
URL: https://github.com/apache/pinot/pull/19996#discussion_r4343154908
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic JSON converter to the value at the given document id. By default, returns
+ /// `jsonTransformer.transform(getValue(docId))`, passing null values to the converter as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param jsonTransformer deterministic converter to the column's JSON representation
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getJsonValue(int docId, ColumnTransformer jsonTransformer)
Review Comment:
Renamed this to `getTransformedValue(int docId, ColumnTransformer transformer)` in 6dee3f4005 and documented it as a generic deterministic transformation hook. The existing test cases now include a non-JSON conversion while retaining null and exception propagation coverage; all 9 tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19996:
URL: https://github.com/apache/pinot/pull/19996#discussion_r4352112906
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
Review Comment:
Clarified in `cc3dd4a04d`: callers must supply a deterministic transformer and must not depend on invocation counts or side effects. This states the caller contract explicitly. All 9 `ColumnReaderTest` cases and scoped precommit checks passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19996:
URL: https://github.com/apache/pinot/pull/19996#discussion_r4353528351
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,27 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ /// The default invokes [ColumnTransformer#transform(Object)] even when [ColumnTransformer#isNoOp()] returns `true`.
+ ///
+ /// Callers must supply a deterministic transformer: repeated transformations of the same value must produce
+ /// equivalent results. Callers must not depend on invocation counts or transformer side effects.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values, reuse immutable conversion results,
+ /// or skip no-op transformations. Overrides must preserve the conversion result, including nulls and conversion
+ /// failures. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getTransformedValue(int docId, ColumnTransformer transformer)
+ throws IOException {
+ return transformer.transform(getValue(docId));
Review Comment:
Addressed in `8f6117acde`: callers must filter no-op transformers during pipeline setup, and the default asserts `!transformer.isNoOp()` before reading values. Removed the wording allowing no-op inputs or overrides to skip them. This PR has no production call sites of the new hook to update.
The assertion-enabled regression rejects no-op inputs before reading or transformation: it fails before the fix, and all 10 tests pass after. With assertions disabled, `isNoOp()` is not evaluated. Scoped formatting, Checkstyle, license checks and Java 11 bytecode validation also pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default JSON conversion hook to ColumnReader [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19996:
URL: https://github.com/apache/pinot/pull/19996#discussion_r4342980939
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic JSON converter to the value at the given document id. By default, returns
+ /// `jsonTransformer.transform(getValue(docId))`, passing null values to the converter as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param jsonTransformer deterministic converter to the column's JSON representation
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getJsonValue(int docId, ColumnTransformer jsonTransformer)
Review Comment:
This hook accepts any `ColumnTransformer` and simply applies it to `getValue(docId)`; neither the signature nor the implementation is JSON-specific. Could we name it `getTransformedValue(int docId, ColumnTransformer transformer)` and document it as a generic deterministic conversion hook? If the extension point is intended specifically for JSON, could we clarify what JSON-specific contract or optimization requires that scope?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19996:
URL: https://github.com/apache/pinot/pull/19996#discussion_r4347022669
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
Review Comment:
The doc calls the transformer `deterministic`, but nothing enforces it and the default works either way. Consider phrasing it as a requirement on callers/overrides so it reads as a contract rather than an assumption.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getTransformedValue(int docId, ColumnTransformer transformer)
Review Comment:
Since `ColumnTransformer` exposes `isNoOp()`, it might be worth a sentence noting that the default deliberately doesn't short-circuit on a no-op transformer. Overrides that cache are free to skip it, but the contract currently leaves that unsaid.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
Review Comment:
The doc calls the transformer `deterministic`, but nothing enforces it and the default works either way. Consider phrasing it as a requirement on callers/overrides so it reads as a contract rather than an assumption.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getTransformedValue(int docId, ColumnTransformer transformer)
Review Comment:
Since `ColumnTransformer` exposes `isNoOp()`, it might be worth a sentence noting that the default deliberately doesn't short-circuit on a no-op transformer. Overrides that cache are free to skip it, but the contract currently leaves that unsaid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19996:
URL: https://github.com/apache/pinot/pull/19996#issuecomment-6330653117
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19996?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 58.93%. Comparing base ([`2877723`](https://app.codecov.io/gh/apache/pinot/commit/2877723da384b783df372fb203fbf9fef0aea740?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6dee3f4`](https://app.codecov.io/gh/apache/pinot/commit/6dee3f40058b11ac39c15c0b1508371cdd7c7556?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19996 +/- ##
=========================================
Coverage 58.93% 58.93%
+ Complexity 7 1 -6
=========================================
Files 2734 2731 -3
Lines 172323 172318 -5
Branches 28424 28424
=========================================
+ Hits 101552 101553 +1
+ Misses 62166 62159 -7
- Partials 8605 8606 +1
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.93% <100.00%> (+<0.01%)` | :arrow_up: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.93% <100.00%> (+<0.01%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.93% <100.00%> (+<0.01%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19996/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.93% <100.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19996?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add a default value transformation hook to ColumnReader [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19996:
URL: https://github.com/apache/pinot/pull/19996#discussion_r4352109736
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getTransformedValue(int docId, ColumnTransformer transformer)
Review Comment:
Clarified in `cc3dd4a04d`: the default invokes `transform(getValue(docId))` even when `isNoOp()` is true; overrides may skip no-op transformations while preserving the conversion contract. All 9 `ColumnReaderTest` cases and scoped precommit checks passed.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
Review Comment:
Clarified in `cc3dd4a04d`: callers must supply a deterministic transformer and must not depend on invocation counts or side effects. This states the caller contract explicitly. All 9 `ColumnReaderTest` cases and scoped precommit checks passed.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/ColumnReader.java:
##########
@@ -188,6 +189,23 @@ boolean isNull(int docId)
Object getValue(int docId)
throws IOException;
+ /// Applies the supplied deterministic transformer to the value at the given document id. By default, returns
+ /// `transformer.transform(getValue(docId))`, passing null values to the transformer as well.
+ ///
+ /// Readers may override this method to avoid materializing repeated raw values or to reuse immutable conversion
+ /// results. Overrides must preserve the conversion result, including nulls and conversion failures; callers must not
+ /// depend on the converter's invocation count. This method does not change [#getValue(int)] or the typed accessors.
+ /// Callers remain responsible for applying this conversion in the correct order relative to other transformations.
+ ///
+ /// @param docId 0-based document id, from 0 (inclusive) to [#getTotalDocs()] (exclusive)
+ /// @param transformer deterministic transformation to apply to the value
+ /// @throws IOException if an I/O error occurs while reading
+ @Nullable
+ default Object getTransformedValue(int docId, ColumnTransformer transformer)
Review Comment:
Clarified in `cc3dd4a04d`: the default invokes `transform(getValue(docId))` even when `isNoOp()` is true; overrides may skip no-op transformations while preserving the conversion contract. All 9 `ColumnReaderTest` cases and scoped precommit checks passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org