You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wolfkill (via GitHub)" <gi...@apache.org> on 2026/05/29 04:09:00 UTC
[PR] Handle Object-returning scalar transform functions [pinot]
wolfkill opened a new pull request, #18628:
URL: https://github.com/apache/pinot/pull/18628
## Summary
- handle scalar transform functions whose Java return type is `Object` without exposing `ColumnDataType.OBJECT` through transform result metadata
- fetch values for scalar function parameters declared as `Object` from child transform functions
- add coverage for `nullIf` with an `Object` result and an `Object` parameter
Fixes #11743
## Tests
- `JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home ./mvnw -pl pinot-core -am -Dtest=ScalarTransformFunctionWrapperTest#testObjectResultScalarTransformFunction+testObjectParameterScalarTransformFunction -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test`
- `JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home ./mvnw -pl pinot-core -am -Dtest=ScalarTransformFunctionWrapperTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Handle Object-returning scalar transform functions [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18628:
URL: https://github.com/apache/pinot/pull/18628#discussion_r3330194066
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -66,7 +66,8 @@ public ScalarTransformFunctionWrapper(FunctionInfo functionInfo) {
ColumnDataType resultType = FunctionUtils.getColumnDataType(resultClass);
if (resultType != null) {
_resultType = resultType;
Review Comment:
This changes every OBJECT-returning scalar UDF to advertise STRING output and stringify the value. That silently changes query semantics for functions like `nullIf`/`caseWhen`: `NULLIF(int_col, 0)` now becomes a VARCHAR result instead of preserving the input type, so downstream arithmetic/comparisons will see the wrong type. We need a type-preserving resolution here rather than coercing OBJECT to STRING.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Handle Object-returning scalar transform functions [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18628:
URL: https://github.com/apache/pinot/pull/18628#discussion_r3331091520
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -482,4 +486,65 @@ private void getNonLiteralValues(ValueBlock valueBlock) {
}
}
}
+
+ private static Object[] transformToObjectValues(TransformFunction transformFunction, ValueBlock valueBlock) {
+ TransformResultMetadata resultMetadata = transformFunction.getResultMetadata();
+ DataType dataType = resultMetadata.getDataType();
+ if (resultMetadata.isSingleValue()) {
+ switch (dataType) {
+ case BOOLEAN: {
+ int[] intValues = transformFunction.transformToIntValuesSV(valueBlock);
+ int numValues = intValues.length;
+ Boolean[] booleanValues = new Boolean[numValues];
+ for (int i = 0; i < numValues; i++) {
+ booleanValues[i] = intValues[i] == 1;
+ }
+ return booleanValues;
+ }
+ case INT:
+ return ArrayUtils.toObject(transformFunction.transformToIntValuesSV(valueBlock));
+ case LONG:
+ return ArrayUtils.toObject(transformFunction.transformToLongValuesSV(valueBlock));
+ case FLOAT:
+ return ArrayUtils.toObject(transformFunction.transformToFloatValuesSV(valueBlock));
+ case DOUBLE:
+ return ArrayUtils.toObject(transformFunction.transformToDoubleValuesSV(valueBlock));
+ case BIG_DECIMAL:
+ return transformFunction.transformToBigDecimalValuesSV(valueBlock);
+ case TIMESTAMP: {
+ long[] longValues = transformFunction.transformToLongValuesSV(valueBlock);
+ int numValues = longValues.length;
+ Timestamp[] timestampValues = new Timestamp[numValues];
+ for (int i = 0; i < numValues; i++) {
+ timestampValues[i] = new Timestamp(longValues[i]);
+ }
+ return timestampValues;
+ }
+ case STRING:
+ return transformFunction.transformToStringValuesSV(valueBlock);
+ case BYTES:
+ return transformFunction.transformToBytesValuesSV(valueBlock);
+ default:
+ throw new IllegalStateException("Unsupported data type: " + dataType);
Review Comment:
JSON isn't handled, so a JSON column or JSON-returning expr passed to an Object param (e.g. `nullIf(jsonCol, '...')`) hits this default and throws — even though JSON is stored as STRING. Can we add `case JSON:` next to STRING (here and in the MV switch)? There's a `jsonSV` test column handy to cover it.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -482,4 +486,65 @@ private void getNonLiteralValues(ValueBlock valueBlock) {
}
}
}
+
+ private static Object[] transformToObjectValues(TransformFunction transformFunction, ValueBlock valueBlock) {
+ TransformResultMetadata resultMetadata = transformFunction.getResultMetadata();
+ DataType dataType = resultMetadata.getDataType();
+ if (resultMetadata.isSingleValue()) {
+ switch (dataType) {
+ case BOOLEAN: {
+ int[] intValues = transformFunction.transformToIntValuesSV(valueBlock);
+ int numValues = intValues.length;
+ Boolean[] booleanValues = new Boolean[numValues];
+ for (int i = 0; i < numValues; i++) {
+ booleanValues[i] = intValues[i] == 1;
+ }
+ return booleanValues;
+ }
+ case INT:
+ return ArrayUtils.toObject(transformFunction.transformToIntValuesSV(valueBlock));
+ case LONG:
+ return ArrayUtils.toObject(transformFunction.transformToLongValuesSV(valueBlock));
+ case FLOAT:
+ return ArrayUtils.toObject(transformFunction.transformToFloatValuesSV(valueBlock));
+ case DOUBLE:
+ return ArrayUtils.toObject(transformFunction.transformToDoubleValuesSV(valueBlock));
+ case BIG_DECIMAL:
+ return transformFunction.transformToBigDecimalValuesSV(valueBlock);
+ case TIMESTAMP: {
+ long[] longValues = transformFunction.transformToLongValuesSV(valueBlock);
+ int numValues = longValues.length;
+ Timestamp[] timestampValues = new Timestamp[numValues];
+ for (int i = 0; i < numValues; i++) {
+ timestampValues[i] = new Timestamp(longValues[i]);
+ }
+ return timestampValues;
+ }
+ case STRING:
+ return transformFunction.transformToStringValuesSV(valueBlock);
+ case BYTES:
+ return transformFunction.transformToBytesValuesSV(valueBlock);
+ default:
+ throw new IllegalStateException("Unsupported data type: " + dataType);
+ }
+ }
+ switch (dataType) {
+ case INT:
+ return transformFunction.transformToIntValuesMV(valueBlock);
+ case LONG:
+ return transformFunction.transformToLongValuesMV(valueBlock);
+ case FLOAT:
+ return transformFunction.transformToFloatValuesMV(valueBlock);
+ case DOUBLE:
+ return transformFunction.transformToDoubleValuesMV(valueBlock);
+ case BIG_DECIMAL:
+ return transformFunction.transformToBigDecimalValuesMV(valueBlock);
+ case STRING:
+ return transformFunction.transformToStringValuesMV(valueBlock);
+ case BYTES:
+ return transformFunction.transformToBytesValuesMV(valueBlock);
+ default:
+ throw new IllegalStateException("Unsupported multi-value data type: " + dataType);
Review Comment:
MV switch is missing BOOLEAN and TIMESTAMP, which the SV switch above handles — an MV boolean/timestamp child feeding an Object param would throw here. Intended, or should it mirror the SV cases?
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -482,4 +486,65 @@ private void getNonLiteralValues(ValueBlock valueBlock) {
}
}
}
+
+ private static Object[] transformToObjectValues(TransformFunction transformFunction, ValueBlock valueBlock) {
Review Comment:
Most of this helper (the numeric/decimal/timestamp/boolean/bytes SV branches and the whole MV switch) isn't exercised by the tests — only the STRING path is. Worth a couple more cases, especially one where the Object function returns null so the null-placeholder path gets hit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Handle Object-returning scalar transform functions [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18628:
URL: https://github.com/apache/pinot/pull/18628#issuecomment-4570742793
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18628?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `21.05263%` with `30 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.80%. Comparing base ([`a762254`](https://app.codecov.io/gh/apache/pinot/commit/a7622546633ca8868bb8125ebac1145ade1beae6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`693e431`](https://app.codecov.io/gh/apache/pinot/commit/693e4314bebdfa6a3e83afc41f5223c2f28fe7f9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18628?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...sform/function/ScalarTransformFunctionWrapper.java](https://app.codecov.io/gh/apache/pinot/pull/18628?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Foperator%2Ftransform%2Ffunction%2FScalarTransformFunctionWrapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vU2NhbGFyVHJhbnNmb3JtRnVuY3Rpb25XcmFwcGVyLmphdmE=) | 21.05% | [28 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18628?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (a762254) and HEAD (693e431). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (a762254) | HEAD (693e431) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration1|1|0|
>|custom-integration1|1|0|
>|integration2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18628 +/- ##
============================================
- Coverage 64.40% 56.80% -7.60%
+ Complexity 1137 1 -1136
============================================
Files 3337 2565 -772
Lines 206069 149137 -56932
Branches 32128 24104 -8024
============================================
- Hits 132710 84724 -47986
+ Misses 62726 57240 -5486
+ Partials 10633 7173 -3460
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-7.60%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-7.60%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-7.59%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-0.02%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18628?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org