You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/09/02 03:10:53 UTC
[GitHub] [pinot] 61yao opened a new pull request, #9327: [Feature] Support Coalesce for Column Names
61yao opened a new pull request, #9327:
URL: https://github.com/apache/pinot/pull/9327
Coalesce takes a list of arguments and returns the first not null value. If all arguments are null, return a null.
This implementations transform all arguments into string and return the first non-null string value. Null is represented as string value "null"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r965302057
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
+1. For the first version, we can check if all the input are numbers or strings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r964607396
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
I think implicitly treating everything as STRING will give confusing semantics imo. If we don't want to support this function on few data types, then throwing exception is better.
Reference - https://docs.snowflake.com/en/sql-reference/functions/coalesce.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
61yao commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r966283357
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
I added a check for argument type. but the return value has to be a string, otherwise we are not able to represent null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r969247733
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
Review Comment:
I don't think we need this check. We may use `getResultMetadata()` to rule out the unsupported types
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
+ "Only numeric value and string are supported in COALESCE.");
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
+ case STRING:
+ return STRING_SV_NO_DICTIONARY_METADATA;
+ case INT:
+ return INT_SV_NO_DICTIONARY_METADATA;
+ case LONG:
+ return LONG_SV_NO_DICTIONARY_METADATA;
+ case BIG_DECIMAL:
+ return BIG_DECIMAL_SV_NO_DICTIONARY_METADATA;
+ default:
+ throw new RuntimeException("Coalesce only supports numerical and string data type");
+ }
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
+ if (_dataType != FieldSpec.DataType.STRING) {
Review Comment:
We want to check `_dataType.getStoredType()`, same for other places
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
+ "Only numeric value and string are supported in COALESCE.");
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
+ case STRING:
+ return STRING_SV_NO_DICTIONARY_METADATA;
+ case INT:
+ return INT_SV_NO_DICTIONARY_METADATA;
+ case LONG:
+ return LONG_SV_NO_DICTIONARY_METADATA;
Review Comment:
`FLOAT` and `DOUBLE` is missing
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
+ "Only numeric value and string are supported in COALESCE.");
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
Review Comment:
We want to switch on `_dataType.getStoredType()` so that logical type such as `BOOLEAN` and `TIMESTAMP` can be supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
61yao commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r966282719
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
Review Comment:
Should be SV. thanks for spotting that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
61yao commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r969473397
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
+ "Only numeric value and string are supported in COALESCE.");
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
Review Comment:
hmm.. this is already store type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
61yao commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r969449064
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
+ "Only numeric value and string are supported in COALESCE.");
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
+ case STRING:
+ return STRING_SV_NO_DICTIONARY_METADATA;
+ case INT:
+ return INT_SV_NO_DICTIONARY_METADATA;
+ case LONG:
+ return LONG_SV_NO_DICTIONARY_METADATA;
+ case BIG_DECIMAL:
+ return BIG_DECIMAL_SV_NO_DICTIONARY_METADATA;
+ default:
+ throw new RuntimeException("Coalesce only supports numerical and string data type");
+ }
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
+ if (_dataType != FieldSpec.DataType.STRING) {
Review Comment:
This is already stored type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9327:
URL: https://github.com/apache/pinot/pull/9327#issuecomment-1235038303
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9327?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9327](https://codecov.io/gh/apache/pinot/pull/9327?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dbc236f) into [master](https://codecov.io/gh/apache/pinot/commit/1c9528c1f54d90a0a62b1795c8785cf5647a676a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1c9528c) will **decrease** coverage by `2.57%`.
> The diff coverage is `84.84%`.
```diff
@@ Coverage Diff @@
## master #9327 +/- ##
============================================
- Coverage 69.73% 67.15% -2.58%
+ Complexity 5017 4856 -161
============================================
Files 1873 1398 -475
Lines 99598 72518 -27080
Branches 15163 11622 -3541
============================================
- Hits 69453 48699 -20754
+ Misses 25217 20298 -4919
+ Partials 4928 3521 -1407
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `67.15% <84.84%> (+0.09%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9327?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...r/transform/function/TransformFunctionFactory.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vVHJhbnNmb3JtRnVuY3Rpb25GYWN0b3J5LmphdmE=) | `85.31% <ø> (-4.13%)` | :arrow_down: |
| [.../transform/function/CoalesceTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vQ29hbGVzY2VUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `84.37% <84.37%> (ø)` | |
| [...e/pinot/common/function/TransformFunctionType.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vVHJhbnNmb3JtRnVuY3Rpb25UeXBlLmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/common/config/NettyConfig.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL05ldHR5Q29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ache/pinot/server/access/AccessControlFactory.java](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL2FjY2Vzcy9BY2Nlc3NDb250cm9sRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [736 more](https://codecov.io/gh/apache/pinot/pull/9327/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r967144101
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
didn't. we already support this in #8264 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r970936547
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform double results based on store type.
+ * @param projectionBlock
+ */
+ private double[] getDoublelTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ double[] results = new double[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ double[][] data = new double[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToDoubleValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (double) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform float results based on store type.
+ * @param projectionBlock
+ */
+ private float[] getFloatTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ float[] results = new float[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ float[][] data = new float[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToFloatValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (float) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
+ case STRING:
Review Comment:
(also use intellij autocomplete to generate the branches will automatically be in that order :-) )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r970936879
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform double results based on store type.
+ * @param projectionBlock
+ */
+ private double[] getDoublelTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ double[] results = new double[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ double[][] data = new double[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToDoubleValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (double) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform float results based on store type.
+ * @param projectionBlock
+ */
+ private float[] getFloatTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ float[] results = new float[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ float[][] data = new float[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToFloatValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (float) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
Review Comment:
:+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr merged pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
walterddr merged PR #9327:
URL: https://github.com/apache/pinot/pull/9327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
61yao commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r969473777
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
Review Comment:
Removed the second check but we need the type to be identifier to get the null bit map?
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
+ "Only numeric value and string are supported in COALESCE.");
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
+ case STRING:
+ return STRING_SV_NO_DICTIONARY_METADATA;
+ case INT:
+ return INT_SV_NO_DICTIONARY_METADATA;
+ case LONG:
+ return LONG_SV_NO_DICTIONARY_METADATA;
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
61yao commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r967561019
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
Updated to use NullValueUtils.getDefaultNullValue() and only supports numeric and string types. Also it requires the type to be same for all arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r967426766
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
@walterddr Not really. We use min value as the default for numeric types except for big decimal, which doesn't have a min value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r970274920
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform double results based on store type.
+ * @param projectionBlock
+ */
+ private double[] getDoublelTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ double[] results = new double[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ double[][] data = new double[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToDoubleValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (double) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform float results based on store type.
+ * @param projectionBlock
+ */
+ private float[] getFloatTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ float[] results = new float[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ float[][] data = new float[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToFloatValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (float) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
+ case STRING:
+ return STRING_SV_NO_DICTIONARY_METADATA;
+ case INT:
+ return INT_SV_NO_DICTIONARY_METADATA;
+ case LONG:
+ return LONG_SV_NO_DICTIONARY_METADATA;
+ case BIG_DECIMAL:
+ return BIG_DECIMAL_SV_NO_DICTIONARY_METADATA;
+ case FLOAT:
+ return FLOAT_SV_NO_DICTIONARY_METADATA;
+ case DOUBLE:
+ return DOUBLE_SV_NO_DICTIONARY_METADATA;
+ default:
+ throw new RuntimeException("Coalesce only supports numerical and string data type");
+ }
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
(minor) Follow the same sequence (`INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`, `STRING`) as the interface for easier tracking
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform double results based on store type.
+ * @param projectionBlock
+ */
+ private double[] getDoublelTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ double[] results = new double[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ double[][] data = new double[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToDoubleValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (double) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform float results based on store type.
+ * @param projectionBlock
+ */
+ private float[] getFloatTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ float[] results = new float[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ float[][] data = new float[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToFloatValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (float) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
+ case STRING:
Review Comment:
(minor) We usually follow the sequence of `INT`, `LONG`, `FLOAT`, `DOUBLE`, `BIG_DECIMAL`, `STRING` (same order as the enum for easier tracking
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform double results based on store type.
+ * @param projectionBlock
+ */
+ private double[] getDoublelTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ double[] results = new double[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ double[][] data = new double[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToDoubleValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (double) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform float results based on store type.
+ * @param projectionBlock
+ */
+ private float[] getFloatTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ float[] results = new float[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ float[][] data = new float[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToFloatValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (float) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
Review Comment:
(minor) Let's calculate the result metadata in the `init()` and store it in a member variable. It has 2 benefits:
1. Fail fast when the type cannot be supported
2. Prevent calculating result metadata multiple times
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform double results based on store type.
+ * @param projectionBlock
+ */
+ private double[] getDoublelTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ double[] results = new double[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ double[][] data = new double[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToDoubleValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (double) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform float results based on store type.
+ * @param projectionBlock
+ */
+ private float[] getFloatTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ float[] results = new float[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ float[][] data = new float[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToFloatValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (float) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
Review Comment:
(minor) Let's change the name to `storedType` to be more explicit
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,361 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform double results based on store type.
+ * @param projectionBlock
+ */
+ private double[] getDoublelTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ double[] results = new double[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ double[][] data = new double[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToDoubleValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (double) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform float results based on store type.
+ * @param projectionBlock
+ */
+ private float[] getFloatTransformResults(ProjectionBlock projectionBlock) {
Review Comment:
(minor) Put this method before `getDoublelTransformResults()` for easier tracking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r964604419
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
Review Comment:
Why is this MV ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r966435928
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
Good point. Currently we don't support `null` values for transform function yet. You may read `TransformBlockValSet.getNullBitmap()`, which count the result as `null` when any argument is `null`. This won't work for `coalesce`.
For now, we may use `NullValueUtils.getDefaultNullValue()` for each data type to present the null. Then we should figure out a way to support the real `null` in transform function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 61yao commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
61yao commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r969472833
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,276 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.NullValueUtils;
+import org.roaringbitmap.RoaringBitmap;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names and single type. The type can be either numeric or string.
+ * Number of arguments has to be greater than 0.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private TransformFunction[] _transformFunctions;
+ private FieldSpec.DataType _dataType;
+
+ /**
+ * Returns a bit map of corresponding column.
+ * Returns an empty bitmap by default if null option is disabled.
+ */
+ private static RoaringBitmap[] getNullBitMaps(ProjectionBlock projectionBlock,
+ TransformFunction[] transformFunctions) {
+ RoaringBitmap[] roaringBitmaps = new RoaringBitmap[transformFunctions.length];
+ for (int i = 0; i < roaringBitmaps.length; i++) {
+ TransformFunction func = transformFunctions[i];
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ RoaringBitmap nullBitmap = projectionBlock.getBlockValueSet(columnName).getNullBitmap();
+ roaringBitmaps[i] = nullBitmap;
+ }
+ return roaringBitmaps;
+ }
+
+ /**
+ * Get transform int results based on store type.
+ * @param projectionBlock
+ */
+ private int[] getIntTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ int[] results = new int[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ int[][] data = new int[width][length];
+ RoaringBitmap filledData = new RoaringBitmap();
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToIntValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (int) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform long results based on store type.
+ * @param projectionBlock
+ */
+ private long[] getLongTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ long[] results = new long[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ long[][] data = new long[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToLongValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (long) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform BigDecimal results based on store type.
+ * @param projectionBlock
+ */
+ private BigDecimal[] getBigDecimalTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ BigDecimal[] results = new BigDecimal[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ BigDecimal[][] data = new BigDecimal[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToBigDecimalValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (BigDecimal) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ /**
+ * Get transform String results based on store type.
+ * @param projectionBlock
+ */
+ private String[] getStringTransformResults(ProjectionBlock projectionBlock) {
+ int length = projectionBlock.getNumDocs();
+ String[] results = new String[length];
+ int width = _transformFunctions.length;
+ RoaringBitmap[] nullBitMaps = getNullBitMaps(projectionBlock, _transformFunctions);
+ String[][] data = new String[width][length];
+ RoaringBitmap filledData = new RoaringBitmap(); // indicates whether certain column has be filled in data.
+ for (int i = 0; i < length; i++) {
+ boolean hasNonNullValue = false;
+ for (int j = 0; j < width; j++) {
+ // Consider value as null only when null option is enabled.
+ if (nullBitMaps[j] != null && nullBitMaps[j].contains(i)) {
+ continue;
+ }
+ if (!filledData.contains(j)) {
+ filledData.add(j);
+ data[j] = _transformFunctions[j].transformToStringValuesSV(projectionBlock);
+ }
+ hasNonNullValue = true;
+ results[i] = data[j][i];
+ break;
+ }
+ if (!hasNonNullValue) {
+ results[i] = (String) NullValueUtils.getDefaultNullValue(_dataType);
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ Preconditions.checkArgument(argSize > 0, "COALESCE needs to have at least one argument.");
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ Preconditions.checkArgument(func instanceof IdentifierTransformFunction,
+ "Only column names are supported in COALESCE.");
+ FieldSpec.DataType dataType = func.getResultMetadata().getDataType().getStoredType();
+ Preconditions.checkArgument(dataType.isNumeric() || dataType == FieldSpec.DataType.STRING,
+ "Only numeric value and string are supported in COALESCE.");
+ if (_dataType == null) {
+ _dataType = dataType;
+ } else {
+ Preconditions.checkArgument(dataType.equals(_dataType), "Argument types have to be the same.");
+ }
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ switch (_dataType) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9327: [Feature] Support Coalesce for Column Names
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9327:
URL: https://github.com/apache/pinot/pull/9327#discussion_r967144101
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/CoalesceTransformFunction.java:
##########
@@ -0,0 +1,108 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.TransformFunctionType;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+
+
+/**
+ * The <code>CoalesceTransformFunction</code> implements the Coalesce operator.
+ *
+ * The results are in String format for first non-null value in the argument list.
+ * If all arguments are null, return a 'null' string.
+ * Note: arguments have to be column names.
+ *
+ * Expected result:
+ * Coalesce(nullColumn, columnA): columnA
+ * Coalesce(columnA, nullColumn): nullColumn
+ * Coalesce(nullColumnA, nullColumnB): "null"
+ *
+ * Note this operator only takes column names for now.
+ * SQL Syntax:
+ * Coalesce(columnA, columnB)
+ */
+public class CoalesceTransformFunction extends BaseTransformFunction {
+ private String[] _results;
+ private NullValueVectorReader[] _nullValueReaders;
+ private TransformFunction[] _transformFunctions;
+
+ @Override
+ public String getName() {
+ return TransformFunctionType.COALESCE.getName();
+ }
+
+ @Override
+ public void init(List<TransformFunction> arguments, Map<String, DataSource> dataSourceMap) {
+ int argSize = arguments.size();
+ _nullValueReaders = new NullValueVectorReader[argSize];
+ _transformFunctions = new TransformFunction[argSize];
+ for (int i = 0; i < argSize; i++) {
+ TransformFunction func = arguments.get(i);
+ if (!(func instanceof IdentifierTransformFunction)) {
+ throw new IllegalArgumentException("Only column names are supported in COALESCE.");
+ }
+ String columnName = ((IdentifierTransformFunction) func).getColumnName();
+ NullValueVectorReader nullValueVectorReader = dataSourceMap.get(columnName).getNullValueVector();
+ _nullValueReaders[i] = nullValueVectorReader;
+ _transformFunctions[i] = func;
+ }
+ }
+
+ @Override
+ public TransformResultMetadata getResultMetadata() {
+ return STRING_MV_NO_DICTIONARY_METADATA;
+ }
+
+ @Override
+ public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
Review Comment:
(edited) represented using default nullvalue probably is not a good idea since all numeric data type uses zero as default null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org