You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wolfkill (via GitHub)" <gi...@apache.org> on 2026/05/29 04:09:00 UTC

[PR] Handle Object-returning scalar transform functions [pinot]

wolfkill opened a new pull request, #18628:
URL: https://github.com/apache/pinot/pull/18628

   ## Summary
   - handle scalar transform functions whose Java return type is `Object` without exposing `ColumnDataType.OBJECT` through transform result metadata
   - fetch values for scalar function parameters declared as `Object` from child transform functions
   - add coverage for `nullIf` with an `Object` result and an `Object` parameter
   
   Fixes #11743
   
   ## Tests
   - `JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home ./mvnw -pl pinot-core -am -Dtest=ScalarTransformFunctionWrapperTest#testObjectResultScalarTransformFunction+testObjectParameterScalarTransformFunction -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test`
   - `JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home ./mvnw -pl pinot-core -am -Dtest=ScalarTransformFunctionWrapperTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test`
   - `git diff --check`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Handle Object-returning scalar transform functions [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18628:
URL: https://github.com/apache/pinot/pull/18628#discussion_r3330194066


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -66,7 +66,8 @@ public ScalarTransformFunctionWrapper(FunctionInfo functionInfo) {
     ColumnDataType resultType = FunctionUtils.getColumnDataType(resultClass);
     if (resultType != null) {
       _resultType = resultType;

Review Comment:
   This changes every OBJECT-returning scalar UDF to advertise STRING output and stringify the value. That silently changes query semantics for functions like `nullIf`/`caseWhen`: `NULLIF(int_col, 0)` now becomes a VARCHAR result instead of preserving the input type, so downstream arithmetic/comparisons will see the wrong type. We need a type-preserving resolution here rather than coercing OBJECT to STRING.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Handle Object-returning scalar transform functions [pinot]

Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18628:
URL: https://github.com/apache/pinot/pull/18628#discussion_r3331091520


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -482,4 +486,65 @@ private void getNonLiteralValues(ValueBlock valueBlock) {
       }
     }
   }
+
+  private static Object[] transformToObjectValues(TransformFunction transformFunction, ValueBlock valueBlock) {
+    TransformResultMetadata resultMetadata = transformFunction.getResultMetadata();
+    DataType dataType = resultMetadata.getDataType();
+    if (resultMetadata.isSingleValue()) {
+      switch (dataType) {
+        case BOOLEAN: {
+          int[] intValues = transformFunction.transformToIntValuesSV(valueBlock);
+          int numValues = intValues.length;
+          Boolean[] booleanValues = new Boolean[numValues];
+          for (int i = 0; i < numValues; i++) {
+            booleanValues[i] = intValues[i] == 1;
+          }
+          return booleanValues;
+        }
+        case INT:
+          return ArrayUtils.toObject(transformFunction.transformToIntValuesSV(valueBlock));
+        case LONG:
+          return ArrayUtils.toObject(transformFunction.transformToLongValuesSV(valueBlock));
+        case FLOAT:
+          return ArrayUtils.toObject(transformFunction.transformToFloatValuesSV(valueBlock));
+        case DOUBLE:
+          return ArrayUtils.toObject(transformFunction.transformToDoubleValuesSV(valueBlock));
+        case BIG_DECIMAL:
+          return transformFunction.transformToBigDecimalValuesSV(valueBlock);
+        case TIMESTAMP: {
+          long[] longValues = transformFunction.transformToLongValuesSV(valueBlock);
+          int numValues = longValues.length;
+          Timestamp[] timestampValues = new Timestamp[numValues];
+          for (int i = 0; i < numValues; i++) {
+            timestampValues[i] = new Timestamp(longValues[i]);
+          }
+          return timestampValues;
+        }
+        case STRING:
+          return transformFunction.transformToStringValuesSV(valueBlock);
+        case BYTES:
+          return transformFunction.transformToBytesValuesSV(valueBlock);
+        default:
+          throw new IllegalStateException("Unsupported data type: " + dataType);

Review Comment:
   JSON isn't handled, so a JSON column or JSON-returning expr passed to an Object param (e.g. `nullIf(jsonCol, '...')`) hits this default and throws — even though JSON is stored as STRING. Can we add `case JSON:` next to STRING (here and in the MV switch)? There's a `jsonSV` test column handy to cover it.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -482,4 +486,65 @@ private void getNonLiteralValues(ValueBlock valueBlock) {
       }
     }
   }
+
+  private static Object[] transformToObjectValues(TransformFunction transformFunction, ValueBlock valueBlock) {
+    TransformResultMetadata resultMetadata = transformFunction.getResultMetadata();
+    DataType dataType = resultMetadata.getDataType();
+    if (resultMetadata.isSingleValue()) {
+      switch (dataType) {
+        case BOOLEAN: {
+          int[] intValues = transformFunction.transformToIntValuesSV(valueBlock);
+          int numValues = intValues.length;
+          Boolean[] booleanValues = new Boolean[numValues];
+          for (int i = 0; i < numValues; i++) {
+            booleanValues[i] = intValues[i] == 1;
+          }
+          return booleanValues;
+        }
+        case INT:
+          return ArrayUtils.toObject(transformFunction.transformToIntValuesSV(valueBlock));
+        case LONG:
+          return ArrayUtils.toObject(transformFunction.transformToLongValuesSV(valueBlock));
+        case FLOAT:
+          return ArrayUtils.toObject(transformFunction.transformToFloatValuesSV(valueBlock));
+        case DOUBLE:
+          return ArrayUtils.toObject(transformFunction.transformToDoubleValuesSV(valueBlock));
+        case BIG_DECIMAL:
+          return transformFunction.transformToBigDecimalValuesSV(valueBlock);
+        case TIMESTAMP: {
+          long[] longValues = transformFunction.transformToLongValuesSV(valueBlock);
+          int numValues = longValues.length;
+          Timestamp[] timestampValues = new Timestamp[numValues];
+          for (int i = 0; i < numValues; i++) {
+            timestampValues[i] = new Timestamp(longValues[i]);
+          }
+          return timestampValues;
+        }
+        case STRING:
+          return transformFunction.transformToStringValuesSV(valueBlock);
+        case BYTES:
+          return transformFunction.transformToBytesValuesSV(valueBlock);
+        default:
+          throw new IllegalStateException("Unsupported data type: " + dataType);
+      }
+    }
+    switch (dataType) {
+      case INT:
+        return transformFunction.transformToIntValuesMV(valueBlock);
+      case LONG:
+        return transformFunction.transformToLongValuesMV(valueBlock);
+      case FLOAT:
+        return transformFunction.transformToFloatValuesMV(valueBlock);
+      case DOUBLE:
+        return transformFunction.transformToDoubleValuesMV(valueBlock);
+      case BIG_DECIMAL:
+        return transformFunction.transformToBigDecimalValuesMV(valueBlock);
+      case STRING:
+        return transformFunction.transformToStringValuesMV(valueBlock);
+      case BYTES:
+        return transformFunction.transformToBytesValuesMV(valueBlock);
+      default:
+        throw new IllegalStateException("Unsupported multi-value data type: " + dataType);

Review Comment:
   MV switch is missing BOOLEAN and TIMESTAMP, which the SV switch above handles — an MV boolean/timestamp child feeding an Object param would throw here. Intended, or should it mirror the SV cases?



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java:
##########
@@ -482,4 +486,65 @@ private void getNonLiteralValues(ValueBlock valueBlock) {
       }
     }
   }
+
+  private static Object[] transformToObjectValues(TransformFunction transformFunction, ValueBlock valueBlock) {

Review Comment:
   Most of this helper (the numeric/decimal/timestamp/boolean/bytes SV branches and the whole MV switch) isn't exercised by the tests — only the STRING path is. Worth a couple more cases, especially one where the Object function returns null so the null-placeholder path gets hit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Handle Object-returning scalar transform functions [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18628:
URL: https://github.com/apache/pinot/pull/18628#issuecomment-4570742793

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18628?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `21.05263%` with `30 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 56.80%. Comparing base ([`a762254`](https://app.codecov.io/gh/apache/pinot/commit/a7622546633ca8868bb8125ebac1145ade1beae6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`693e431`](https://app.codecov.io/gh/apache/pinot/commit/693e4314bebdfa6a3e83afc41f5223c2f28fe7f9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18628?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...sform/function/ScalarTransformFunctionWrapper.java](https://app.codecov.io/gh/apache/pinot/pull/18628?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Foperator%2Ftransform%2Ffunction%2FScalarTransformFunctionWrapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vU2NhbGFyVHJhbnNmb3JtRnVuY3Rpb25XcmFwcGVyLmphdmE=) | 21.05% | [28 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18628?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (a762254) and HEAD (693e431). Click for more details.
   > 
   > <details><summary>HEAD has 16 uploads less than BASE</summary>
   >
   >| Flag | BASE (a762254) | HEAD (693e431) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests|2|1|
   >|temurin|5|1|
   >|unittests2|1|0|
   >|integration|3|0|
   >|integration1|1|0|
   >|custom-integration1|1|0|
   >|integration2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18628      +/-   ##
   ============================================
   - Coverage     64.40%   56.80%   -7.60%     
   + Complexity     1137        1    -1136     
   ============================================
     Files          3337     2565     -772     
     Lines        206069   149137   -56932     
     Branches      32128    24104    -8024     
   ============================================
   - Hits         132710    84724   -47986     
   + Misses        62726    57240    -5486     
   + Partials      10633     7173    -3460     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-7.60%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-7.60%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-7.59%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <21.05%> (-0.02%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18628/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18628?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org