You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/29 17:23:32 UTC
[GitHub] [pinot] somandal opened a new pull request, #8993: Add support for multi-value noDict column types for group-by and filter type queries
somandal opened a new pull request, #8993:
URL: https://github.com/apache/pinot/pull/8993
This part 2 of a multi-part set of changes to add support for multi-value noDict column types for immutable and mutable fixed width columns. Support to create segments with multi-value columns with dictionary disabled already existed. On trying to create a multi-value RAW column and querying it, we ran into this issue: https://github.com/apache/pinot/issues/8875
Original PR that adds complete support for querying MV raw columns - https://github.com/apache/pinot/pull/8917
This PR adds support for group-by and filter type queries by modifying the `MVScanDocIdIterator` and `NoDictionarySingleColumnGroupKeyGenerator` to add support for multi-value noDict column types. With this change group-by, filter, etc type queries can be supported. Additional changes in this PR add test cases to ensure that querying multi-value noDict columns work.
Part 1 can be found here: https://github.com/apache/pinot/pull/8953
cc @siddharthteotia @Jackie-Jiang @walterddr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910687228
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
+ {
+ // Test a select with filter query on a MV raw column identifier
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertEquals(intVal[0], i % 5);
+ assertEquals(intVal[1], (i % 5) + MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) i % 5);
+ assertEquals(doubleVal[1], (double) (i % 5) + MV_OFFSET);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), i % 5);
+ assertEquals(Integer.parseInt(stringVal[1]), (i % 5) + MV_OFFSET);
+ }
+ }
+ {
+ // Test a select with filter query (OR) on two MV raw column identifiers (int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "OR mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ double[] doubleVal = (double[]) values[1];
+ assertTrue(intVal[0] < 5 || intVal[1] < 5 || doubleVal[0] > 1104.0 || doubleVal[1] > 1104.0);
+ }
+ }
+ {
+ // Test a select with filter query on a long MV raw column identifier
+ String query = "SELECT mvRawLongCol from testTable where mvRawLongCol > 1100 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ long[] longVal = (long[]) values[0];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] > 1100 || longVal[1] > 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on a string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol = '1100' LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol = '1100' AND mvRawFloatCol = '1100.0' AND mvRawLongCol = '1100' AND mvRawDoubleCol = '1100.0' "
+ + "LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0);
+ }
+ }
+ {
+ // Test a select with filter query (AND) on two MV raw column identifiers (one int and another double) such that
+ // the values in the filter are mutually exclusive
+ // No match should be found
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "AND mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 0);
+ }
+ {
+ // Test a select with filter IN query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol IN (1100, 1101) AND mvRawFloatCol IN (1100.0, 1101.0) AND mvRawLongCol "
+ + "IN (1100, 1101) AND mvRawDoubleCol IN (1100.0, 1101.0) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100 || intVal[0] == 1101 || intVal[1] == 1101);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F || floatVal[0] == 1101.0F
+ || floatVal[1] == 1101.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L || longVal[0] == 1101L || longVal[1] == 1101L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0 || doubleVal[0] == 1101.0
+ || doubleVal[1] == 1101.0);
+ }
+ }
+ {
+ // Test a select with filter IN query on the string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol IN ('1100', '1101') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100
+ || Integer.parseInt(stringVal[0]) == 1101 || Integer.parseInt(stringVal[1]) == 1101);
+ }
+ }
+ {
+ // Test a select with filter query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawIntCol) < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter = query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawDoubleCol) = 2 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter IN query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawStringCol) IN (2, 5) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ }
+
Review Comment:
For the next test on `testSimpleAggregateQueries`, can we add 1/2 queries with nesting a transform inside the aggregate if that is not already covered elsewhere ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910711631
##########
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/NoDictionarySingleColumnGroupKeyGenerator.java:
##########
@@ -164,8 +166,113 @@ private Map createGroupKeyMap(DataType keyType) {
@Override
public void generateKeysForBlock(TransformBlock transformBlock, int[][] groupKeys) {
Review Comment:
Shouldn't we make the same enhancement to `NoDictionaryMultiColumnGroupKeyGenerator` as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911266357
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911314611
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -468,6 +907,80 @@ private void validateSimpleAggregateQueryResults(ResultTable resultTable, DataSc
assertEquals(avgInt, avgIntRaw);
}
+ @Test
+ public void testAggregateWithFilterQueries() {
+ {
+ // Aggregation on int columns with filter
+ String query = "SELECT COUNTMV(mvIntCol), COUNTMV(mvRawIntCol), SUMMV(mvIntCol), SUMMV(mvRawIntCol), "
+ + "MINMV(mvIntCol), MINMV(mvRawIntCol), MAXMV(mvIntCol), MAXMV(mvRawIntCol), AVGMV(mvIntCol), "
+ + "AVGMV(mvRawIntCol) from testTable WHERE mvRawIntCol > 1000";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvIntCol)", "countmv(mvRawIntCol)", "summv(mvIntCol)", "summv(mvRawIntCol)", "minmv(mvIntCol)",
+ "minmv(mvRawIntCol)", "maxmv(mvIntCol)", "maxmv(mvRawIntCol)", "avgmv(mvIntCol)", "avgmv(mvRawIntCol)"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE
+ });
+ validateAggregateWithFilterQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on double columns with filter
+ String query = "SELECT COUNTMV(mvDoubleCol), COUNTMV(mvRawDoubleCol), SUMMV(mvDoubleCol), SUMMV(mvRawDoubleCol), "
+ + "MINMV(mvDoubleCol), MINMV(mvRawDoubleCol), MAXMV(mvDoubleCol), MAXMV(mvRawDoubleCol), AVGMV(mvDoubleCol), "
+ + "AVGMV(mvRawDoubleCol) from testTable WHERE mvRawDoubleCol > 1000.0";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvDoubleCol)", "countmv(mvRawDoubleCol)", "summv(mvDoubleCol)", "summv(mvRawDoubleCol)",
+ "minmv(mvDoubleCol)", "minmv(mvRawDoubleCol)", "maxmv(mvDoubleCol)", "maxmv(mvRawDoubleCol)",
+ "avgmv(mvDoubleCol)", "avgmv(mvRawDoubleCol)"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE
+ });
+ validateAggregateWithFilterQueryResults(resultTable, dataSchema);
+ }
+ }
+
+ private void validateAggregateWithFilterQueryResults(ResultTable resultTable, DataSchema expectedDataSchema) {
+ assertNotNull(resultTable);
+ assertEquals(resultTable.getDataSchema(), expectedDataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+
+ Object[] values = recordRows.get(0);
+ long count = (long) values[0];
+ long countRaw = (long) values[1];
+ assertEquals(count, 80);
+ assertEquals(count, countRaw);
+
+ double sum = (double) values[2];
+ double sumRaw = (double) values[3];
+ assertEquals(sum, 84360.0);
+ assertEquals(sum, sumRaw);
+
+ double min = (double) values[4];
+ double minRaw = (double) values[5];
+ assertEquals(min, 1000.0);
+ assertEquals(min, minRaw);
+
+ double max = (double) values[6];
+ double maxRaw = (double) values[7];
+ assertEquals(max, 1109.0);
+ assertEquals(max, maxRaw);
+
+ double avg = (double) values[8];
+ double avgRaw = (double) values[9];
+ assertEquals(avg, 1054.5);
+ assertEquals(avg, avgRaw);
+ }
+
@Test
public void testAggregateWithGroupByQueries() {
Review Comment:
added tests for the first 3 cases as discussed. the others are already covered. Transform is also covered in a different test function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910684831
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
+ {
+ // Test a select with filter query on a MV raw column identifier
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertEquals(intVal[0], i % 5);
+ assertEquals(intVal[1], (i % 5) + MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) i % 5);
+ assertEquals(doubleVal[1], (double) (i % 5) + MV_OFFSET);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), i % 5);
+ assertEquals(Integer.parseInt(stringVal[1]), (i % 5) + MV_OFFSET);
+ }
+ }
+ {
+ // Test a select with filter query (OR) on two MV raw column identifiers (int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "OR mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ double[] doubleVal = (double[]) values[1];
+ assertTrue(intVal[0] < 5 || intVal[1] < 5 || doubleVal[0] > 1104.0 || doubleVal[1] > 1104.0);
+ }
+ }
+ {
+ // Test a select with filter query on a long MV raw column identifier
+ String query = "SELECT mvRawLongCol from testTable where mvRawLongCol > 1100 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ long[] longVal = (long[]) values[0];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] > 1100 || longVal[1] > 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on a string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol = '1100' LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol = '1100' AND mvRawFloatCol = '1100.0' AND mvRawLongCol = '1100' AND mvRawDoubleCol = '1100.0' "
+ + "LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0);
+ }
+ }
+ {
+ // Test a select with filter query (AND) on two MV raw column identifiers (one int and another double) such that
+ // the values in the filter are mutually exclusive
+ // No match should be found
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "AND mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 0);
+ }
+ {
+ // Test a select with filter IN query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol IN (1100, 1101) AND mvRawFloatCol IN (1100.0, 1101.0) AND mvRawLongCol "
+ + "IN (1100, 1101) AND mvRawDoubleCol IN (1100.0, 1101.0) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100 || intVal[0] == 1101 || intVal[1] == 1101);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F || floatVal[0] == 1101.0F
+ || floatVal[1] == 1101.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L || longVal[0] == 1101L || longVal[1] == 1101L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0 || doubleVal[0] == 1101.0
+ || doubleVal[1] == 1101.0);
+ }
+ }
+ {
+ // Test a select with filter IN query on the string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol IN ('1100', '1101') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100
+ || Integer.parseInt(stringVal[0]) == 1101 || Integer.parseInt(stringVal[1]) == 1101);
+ }
+ }
+ {
+ // Test a select with filter query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawIntCol) < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter = query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawDoubleCol) = 2 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter IN query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawStringCol) IN (2, 5) LIMIT 10";
Review Comment:
Is ORDER BY not needed to make the test result deterministic ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911302896
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
Review Comment:
Modified this to add order by for all columns and then added validations for explicit values in each row. thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia merged pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia merged PR #8993:
URL: https://github.com/apache/pinot/pull/8993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910678999
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
Review Comment:
Does this exercise the scenario I had talked about offline ? Some rows in your generated data should have MV column row / array with value 0 or something else for VALUEIN to remove them during transform ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910683733
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
Review Comment:
Can we add a transform function (e.g ARRAYLENGTH) to one of the queries here and also add it to GROUP BY ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910677514
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
Review Comment:
This seems to contradict the TODO mentioned at line 344 regarding ORDER BY support for MV ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8993:
URL: https://github.com/apache/pinot/pull/8993#issuecomment-1170329642
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8993?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8993](https://codecov.io/gh/apache/pinot/pull/8993?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (94fff8a) into [master](https://codecov.io/gh/apache/pinot/commit/4152f60a849fe38f57b95600570555a5522c0c4c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4152f60) will **increase** coverage by `0.22%`.
> The diff coverage is `46.72%`.
```diff
@@ Coverage Diff @@
## master #8993 +/- ##
============================================
+ Coverage 63.07% 63.30% +0.22%
- Complexity 4714 4719 +5
============================================
Files 1773 1773
Lines 93315 93429 +114
Branches 14040 14060 +20
============================================
+ Hits 58861 59146 +285
+ Misses 30209 30047 -162
+ Partials 4245 4236 -9
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `66.66% <46.72%> (+0.37%)` | :arrow_up: |
| unittests2 | `14.96% <0.00%> (-0.08%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8993?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pby/NoDictionarySingleColumnGroupKeyGenerator.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9ncm91cGJ5L05vRGljdGlvbmFyeVNpbmdsZUNvbHVtbkdyb3VwS2V5R2VuZXJhdG9yLmphdmE=) | `70.58% <26.38%> (-17.97%)` | :arrow_down: |
| [...e/operator/dociditerators/MVScanDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9NVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `67.94% <76.00%> (+17.94%)` | :arrow_up: |
| [...ller/validation/OfflineSegmentIntervalChecker.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL09mZmxpbmVTZWdtZW50SW50ZXJ2YWxDaGVja2VyLmphdmE=) | `25.84% <0.00%> (-34.84%)` | :arrow_down: |
| [...roller/helix/core/relocation/SegmentRelocator.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlbG9jYXRpb24vU2VnbWVudFJlbG9jYXRvci5qYXZh) | `13.51% <0.00%> (-18.92%)` | :arrow_down: |
| [.../helix/core/minion/MinionInstancesCleanupTask.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9NaW5pb25JbnN0YW5jZXNDbGVhbnVwVGFzay5qYXZh) | `81.81% <0.00%> (-9.10%)` | :arrow_down: |
| [...troller/helix/core/retention/RetentionManager.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JldGVudGlvbi9SZXRlbnRpb25NYW5hZ2VyLmphdmE=) | `77.41% <0.00%> (-7.26%)` | :arrow_down: |
| [...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `75.91% <0.00%> (-1.58%)` | :arrow_down: |
| [...ces/PinotSegmentUploadDownloadRestletResource.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90U2VnbWVudFVwbG9hZERvd25sb2FkUmVzdGxldFJlc291cmNlLmphdmE=) | `40.83% <0.00%> (-1.25%)` | :arrow_down: |
| [...not/broker/broker/helix/ClusterChangeMediator.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0NsdXN0ZXJDaGFuZ2VNZWRpYXRvci5qYXZh) | `74.19% <0.00%> (-1.08%)` | :arrow_down: |
| [...org/apache/pinot/core/common/ObjectSerDeUtils.java](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vT2JqZWN0U2VyRGVVdGlscy5qYXZh) | `89.63% <0.00%> (-0.82%)` | :arrow_down: |
| ... and [39 more](https://codecov.io/gh/apache/pinot/pull/8993/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8993?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8993?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [4152f60...94fff8a](https://codecov.io/gh/apache/pinot/pull/8993?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911275573
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
+ {
+ // Test a select with filter query on a MV raw column identifier
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertEquals(intVal[0], i % 5);
+ assertEquals(intVal[1], (i % 5) + MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) i % 5);
+ assertEquals(doubleVal[1], (double) (i % 5) + MV_OFFSET);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), i % 5);
+ assertEquals(Integer.parseInt(stringVal[1]), (i % 5) + MV_OFFSET);
+ }
+ }
+ {
+ // Test a select with filter query (OR) on two MV raw column identifiers (int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "OR mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ double[] doubleVal = (double[]) values[1];
+ assertTrue(intVal[0] < 5 || intVal[1] < 5 || doubleVal[0] > 1104.0 || doubleVal[1] > 1104.0);
+ }
+ }
+ {
+ // Test a select with filter query on a long MV raw column identifier
+ String query = "SELECT mvRawLongCol from testTable where mvRawLongCol > 1100 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ long[] longVal = (long[]) values[0];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] > 1100 || longVal[1] > 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on a string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol = '1100' LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol = '1100' AND mvRawFloatCol = '1100.0' AND mvRawLongCol = '1100' AND mvRawDoubleCol = '1100.0' "
+ + "LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0);
+ }
+ }
+ {
+ // Test a select with filter query (AND) on two MV raw column identifiers (one int and another double) such that
+ // the values in the filter are mutually exclusive
+ // No match should be found
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "AND mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 0);
+ }
+ {
+ // Test a select with filter IN query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol IN (1100, 1101) AND mvRawFloatCol IN (1100.0, 1101.0) AND mvRawLongCol "
+ + "IN (1100, 1101) AND mvRawDoubleCol IN (1100.0, 1101.0) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100 || intVal[0] == 1101 || intVal[1] == 1101);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F || floatVal[0] == 1101.0F
+ || floatVal[1] == 1101.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L || longVal[0] == 1101L || longVal[1] == 1101L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0 || doubleVal[0] == 1101.0
+ || doubleVal[1] == 1101.0);
+ }
+ }
+ {
+ // Test a select with filter IN query on the string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol IN ('1100', '1101') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100
+ || Integer.parseInt(stringVal[0]) == 1101 || Integer.parseInt(stringVal[1]) == 1101);
+ }
+ }
+ {
+ // Test a select with filter query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawIntCol) < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter = query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawDoubleCol) = 2 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter IN query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawStringCol) IN (2, 5) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ }
+
Review Comment:
as discussed this is already covered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911260876
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
Review Comment:
updated the comment in the above that ORDER BY is for selection queries and only queries where ORDER BY is present without GROUP BY.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910692036
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -468,6 +907,80 @@ private void validateSimpleAggregateQueryResults(ResultTable resultTable, DataSc
assertEquals(avgInt, avgIntRaw);
}
+ @Test
+ public void testAggregateWithFilterQueries() {
+ {
+ // Aggregation on int columns with filter
+ String query = "SELECT COUNTMV(mvIntCol), COUNTMV(mvRawIntCol), SUMMV(mvIntCol), SUMMV(mvRawIntCol), "
+ + "MINMV(mvIntCol), MINMV(mvRawIntCol), MAXMV(mvIntCol), MAXMV(mvRawIntCol), AVGMV(mvIntCol), "
+ + "AVGMV(mvRawIntCol) from testTable WHERE mvRawIntCol > 1000";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvIntCol)", "countmv(mvRawIntCol)", "summv(mvIntCol)", "summv(mvRawIntCol)", "minmv(mvIntCol)",
+ "minmv(mvRawIntCol)", "maxmv(mvIntCol)", "maxmv(mvRawIntCol)", "avgmv(mvIntCol)", "avgmv(mvRawIntCol)"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE
+ });
+ validateAggregateWithFilterQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on double columns with filter
+ String query = "SELECT COUNTMV(mvDoubleCol), COUNTMV(mvRawDoubleCol), SUMMV(mvDoubleCol), SUMMV(mvRawDoubleCol), "
+ + "MINMV(mvDoubleCol), MINMV(mvRawDoubleCol), MAXMV(mvDoubleCol), MAXMV(mvRawDoubleCol), AVGMV(mvDoubleCol), "
+ + "AVGMV(mvRawDoubleCol) from testTable WHERE mvRawDoubleCol > 1000.0";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvDoubleCol)", "countmv(mvRawDoubleCol)", "summv(mvDoubleCol)", "summv(mvRawDoubleCol)",
+ "minmv(mvDoubleCol)", "minmv(mvRawDoubleCol)", "maxmv(mvDoubleCol)", "maxmv(mvRawDoubleCol)",
+ "avgmv(mvDoubleCol)", "avgmv(mvRawDoubleCol)"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE
+ });
+ validateAggregateWithFilterQueryResults(resultTable, dataSchema);
+ }
+ }
+
+ private void validateAggregateWithFilterQueryResults(ResultTable resultTable, DataSchema expectedDataSchema) {
+ assertNotNull(resultTable);
+ assertEquals(resultTable.getDataSchema(), expectedDataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+
+ Object[] values = recordRows.get(0);
+ long count = (long) values[0];
+ long countRaw = (long) values[1];
+ assertEquals(count, 80);
+ assertEquals(count, countRaw);
+
+ double sum = (double) values[2];
+ double sumRaw = (double) values[3];
+ assertEquals(sum, 84360.0);
+ assertEquals(sum, sumRaw);
+
+ double min = (double) values[4];
+ double minRaw = (double) values[5];
+ assertEquals(min, 1000.0);
+ assertEquals(min, minRaw);
+
+ double max = (double) values[6];
+ double maxRaw = (double) values[7];
+ assertEquals(max, 1109.0);
+ assertEquals(max, maxRaw);
+
+ double avg = (double) values[8];
+ double avgRaw = (double) values[9];
+ assertEquals(avg, 1054.5);
+ assertEquals(avg, avgRaw);
+ }
+
@Test
public void testAggregateWithGroupByQueries() {
Review Comment:
Have we covered following scenarios ?
- single GROUP BY MV column (raw)
- 2 or more GROUP BY MV columns (all raw)
- 2 or more GROUP BY MV columns (mix of raw and dict)
- 2 or more GROUP BY MV columns (all raw) along 1 or more GROUP BY SV column (dict)
- 2 or more GROUP BY MV columns (mix of raw and dict) along 1 or more GROUP BY SV column (raw or dict)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910680710
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
Review Comment:
Let's ensure that we have covered all the MV column types and the filter operators =, IN, RANGE (<, > <=, >=), !=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910675440
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
Review Comment:
Curious if this is enough to ensure the correctness of the values in the selected MV columns in the ResultTable ? Should we not assert / check for those columns as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911172129
##########
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/NoDictionarySingleColumnGroupKeyGenerator.java:
##########
@@ -164,8 +166,113 @@ private Map createGroupKeyMap(DataType keyType) {
@Override
public void generateKeysForBlock(TransformBlock transformBlock, int[][] groupKeys) {
Review Comment:
This function is already implemented for `NoDictionaryMultiColumnGroupKeyGenerator`.
See this link: https://github.com/apache/pinot/blob/1b2c278daefc3f4c9d085094217ae9de469f904f/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/NoDictionaryMultiColumnGroupKeyGenerator.java#L155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911219349
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
Review Comment:
I have a query like this:
```
"SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ "VALUEIN(mvRawIntCol, '0') LIMIT 10"
```
Will that work? Or would you like me to add one on `ARRAYLENGTH` specifically?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911275199
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
+ {
+ // Test a select with filter query on a MV raw column identifier
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertEquals(intVal[0], i % 5);
+ assertEquals(intVal[1], (i % 5) + MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) i % 5);
+ assertEquals(doubleVal[1], (double) (i % 5) + MV_OFFSET);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), i % 5);
+ assertEquals(Integer.parseInt(stringVal[1]), (i % 5) + MV_OFFSET);
+ }
+ }
+ {
+ // Test a select with filter query (OR) on two MV raw column identifiers (int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "OR mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ double[] doubleVal = (double[]) values[1];
+ assertTrue(intVal[0] < 5 || intVal[1] < 5 || doubleVal[0] > 1104.0 || doubleVal[1] > 1104.0);
+ }
+ }
+ {
+ // Test a select with filter query on a long MV raw column identifier
+ String query = "SELECT mvRawLongCol from testTable where mvRawLongCol > 1100 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ long[] longVal = (long[]) values[0];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] > 1100 || longVal[1] > 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on a string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol = '1100' LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol = '1100' AND mvRawFloatCol = '1100.0' AND mvRawLongCol = '1100' AND mvRawDoubleCol = '1100.0' "
+ + "LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0);
+ }
+ }
+ {
+ // Test a select with filter query (AND) on two MV raw column identifiers (one int and another double) such that
+ // the values in the filter are mutually exclusive
+ // No match should be found
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "AND mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 0);
+ }
+ {
+ // Test a select with filter IN query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol IN (1100, 1101) AND mvRawFloatCol IN (1100.0, 1101.0) AND mvRawLongCol "
+ + "IN (1100, 1101) AND mvRawDoubleCol IN (1100.0, 1101.0) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100 || intVal[0] == 1101 || intVal[1] == 1101);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F || floatVal[0] == 1101.0F
+ || floatVal[1] == 1101.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L || longVal[0] == 1101L || longVal[1] == 1101L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0 || doubleVal[0] == 1101.0
+ || doubleVal[1] == 1101.0);
+ }
+ }
+ {
+ // Test a select with filter IN query on the string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol IN ('1100', '1101') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100
+ || Integer.parseInt(stringVal[0]) == 1101 || Integer.parseInt(stringVal[1]) == 1101);
+ }
+ }
+ {
+ // Test a select with filter query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawIntCol) < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter = query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawDoubleCol) = 2 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter IN query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawStringCol) IN (2, 5) LIMIT 10";
Review Comment:
my initial checks only validated that the values across rows are as expected without validating explicitly what those values should be. I did update this to add an order by so that I could also validate explicit values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911266357
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
Review Comment:
done. Modified some queries to use the >=, <= (which were missing), and added one for !=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910687228
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
+ {
+ // Test a select with filter query on a MV raw column identifier
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertEquals(intVal[0], i % 5);
+ assertEquals(intVal[1], (i % 5) + MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) i % 5);
+ assertEquals(doubleVal[1], (double) (i % 5) + MV_OFFSET);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), i % 5);
+ assertEquals(Integer.parseInt(stringVal[1]), (i % 5) + MV_OFFSET);
+ }
+ }
+ {
+ // Test a select with filter query (OR) on two MV raw column identifiers (int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "OR mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ double[] doubleVal = (double[]) values[1];
+ assertTrue(intVal[0] < 5 || intVal[1] < 5 || doubleVal[0] > 1104.0 || doubleVal[1] > 1104.0);
+ }
+ }
+ {
+ // Test a select with filter query on a long MV raw column identifier
+ String query = "SELECT mvRawLongCol from testTable where mvRawLongCol > 1100 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ long[] longVal = (long[]) values[0];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] > 1100 || longVal[1] > 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on a string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol = '1100' LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol = '1100' AND mvRawFloatCol = '1100.0' AND mvRawLongCol = '1100' AND mvRawDoubleCol = '1100.0' "
+ + "LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 4);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0);
+ }
+ }
+ {
+ // Test a select with filter query (AND) on two MV raw column identifiers (one int and another double) such that
+ // the values in the filter are mutually exclusive
+ // No match should be found
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "AND mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 0);
+ }
+ {
+ // Test a select with filter IN query on int, float, long, and double MV raw column identifiers
+ String query = "SELECT mvRawIntCol, mvRawFloatCol, mvRawLongCol, mvRawDoubleCol from testTable where "
+ + "mvRawIntCol IN (1100, 1101) AND mvRawFloatCol IN (1100.0, 1101.0) AND mvRawLongCol "
+ + "IN (1100, 1101) AND mvRawDoubleCol IN (1100.0, 1101.0) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawFloatCol", "mvRawLongCol", "mvRawDoubleCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.FLOAT_ARRAY,
+ DataSchema.ColumnDataType.LONG_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertTrue(intVal[0] == 1100 || intVal[1] == 1100 || intVal[0] == 1101 || intVal[1] == 1101);
+
+ float[] floatVal = (float[]) values[1];
+ assertEquals(floatVal[1] - floatVal[0], (float) MV_OFFSET);
+ assertTrue(floatVal[0] == 1100.0F || floatVal[1] == 1100.0F || floatVal[0] == 1101.0F
+ || floatVal[1] == 1101.0F);
+
+ long[] longVal = (long[]) values[2];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] == 1100L || longVal[1] == 1100L || longVal[0] == 1101L || longVal[1] == 1101L);
+
+ double[] doubleVal = (double[]) values[3];
+ assertEquals(doubleVal[1] - doubleVal[0], (double) MV_OFFSET);
+ assertTrue(doubleVal[0] == 1100.0 || doubleVal[1] == 1100.0 || doubleVal[0] == 1101.0
+ || doubleVal[1] == 1101.0);
+ }
+ }
+ {
+ // Test a select with filter IN query on the string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol IN ('1100', '1101') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 8);
+
+ for (int i = 0; i < 4; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ String[] stringVal = (String[]) values[0];
+ assertEquals(Integer.parseInt(stringVal[1]) - Integer.parseInt(stringVal[0]), MV_OFFSET);
+ assertTrue(Integer.parseInt(stringVal[0]) == 1100 || Integer.parseInt(stringVal[1]) == 1100
+ || Integer.parseInt(stringVal[0]) == 1101 || Integer.parseInt(stringVal[1]) == 1101);
+ }
+ }
+ {
+ // Test a select with filter query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawIntCol) < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter = query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawDoubleCol) = 2 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ {
+ // Test a select with filter IN query on an arraylength transform function
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where "
+ + "ARRAYLENGTH(mvRawStringCol) IN (2, 5) LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) intVal[0]);
+ assertEquals(doubleVal[1], (double) intVal[1]);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), intVal[0]);
+ assertEquals(Integer.parseInt(stringVal[1]), intVal[1]);
+ }
+ }
+ }
+
Review Comment:
For the next test on testSimpleAggregateQueries, can we add 1/2 queries with nesting a transform inside the aggregate if that is not already covered elsewhere ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910694202
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -468,6 +907,80 @@ private void validateSimpleAggregateQueryResults(ResultTable resultTable, DataSc
assertEquals(avgInt, avgIntRaw);
}
+ @Test
+ public void testAggregateWithFilterQueries() {
+ {
+ // Aggregation on int columns with filter
+ String query = "SELECT COUNTMV(mvIntCol), COUNTMV(mvRawIntCol), SUMMV(mvIntCol), SUMMV(mvRawIntCol), "
+ + "MINMV(mvIntCol), MINMV(mvRawIntCol), MAXMV(mvIntCol), MAXMV(mvRawIntCol), AVGMV(mvIntCol), "
+ + "AVGMV(mvRawIntCol) from testTable WHERE mvRawIntCol > 1000";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvIntCol)", "countmv(mvRawIntCol)", "summv(mvIntCol)", "summv(mvRawIntCol)", "minmv(mvIntCol)",
+ "minmv(mvRawIntCol)", "maxmv(mvIntCol)", "maxmv(mvRawIntCol)", "avgmv(mvIntCol)", "avgmv(mvRawIntCol)"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE
+ });
+ validateAggregateWithFilterQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on double columns with filter
+ String query = "SELECT COUNTMV(mvDoubleCol), COUNTMV(mvRawDoubleCol), SUMMV(mvDoubleCol), SUMMV(mvRawDoubleCol), "
+ + "MINMV(mvDoubleCol), MINMV(mvRawDoubleCol), MAXMV(mvDoubleCol), MAXMV(mvRawDoubleCol), AVGMV(mvDoubleCol), "
+ + "AVGMV(mvRawDoubleCol) from testTable WHERE mvRawDoubleCol > 1000.0";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvDoubleCol)", "countmv(mvRawDoubleCol)", "summv(mvDoubleCol)", "summv(mvRawDoubleCol)",
+ "minmv(mvDoubleCol)", "minmv(mvRawDoubleCol)", "maxmv(mvDoubleCol)", "maxmv(mvRawDoubleCol)",
+ "avgmv(mvDoubleCol)", "avgmv(mvRawDoubleCol)"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE
+ });
+ validateAggregateWithFilterQueryResults(resultTable, dataSchema);
+ }
+ }
+
+ private void validateAggregateWithFilterQueryResults(ResultTable resultTable, DataSchema expectedDataSchema) {
+ assertNotNull(resultTable);
+ assertEquals(resultTable.getDataSchema(), expectedDataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+
+ Object[] values = recordRows.get(0);
+ long count = (long) values[0];
+ long countRaw = (long) values[1];
+ assertEquals(count, 80);
+ assertEquals(count, countRaw);
+
+ double sum = (double) values[2];
+ double sumRaw = (double) values[3];
+ assertEquals(sum, 84360.0);
+ assertEquals(sum, sumRaw);
+
+ double min = (double) values[4];
+ double minRaw = (double) values[5];
+ assertEquals(min, 1000.0);
+ assertEquals(min, minRaw);
+
+ double max = (double) values[6];
+ double maxRaw = (double) values[7];
+ assertEquals(max, 1109.0);
+ assertEquals(max, maxRaw);
+
+ double avg = (double) values[8];
+ double avgRaw = (double) values[9];
+ assertEquals(avg, 1054.5);
+ assertEquals(avg, avgRaw);
+ }
+
@Test
public void testAggregateWithGroupByQueries() {
Review Comment:
Also, can we add transform (on the MV raw group by column) to one of the aggregate group by queries ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911219349
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
Review Comment:
I have a query like this:
```
"SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ "VALUEIN(mvRawIntCol, '0') LIMIT 10"
```
Will that work? Or would you like me to add one on `ARRAYLENGTH` specifically?
update: added one as I thought it may just be useful for testing this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910680710
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
Review Comment:
As long as we have covered all the Mv column types and the filter operators =, IN, RANGE (<, > <=, >=), !=, this should be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910670825
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java:
##########
@@ -85,9 +90,12 @@ public MutableRoaringBitmap applyAnd(ImmutableRoaringBitmap docIds) {
int limit = docIdIterator.nextBatch(buffer);
for (int i = 0; i < limit; i++) {
int nextDocId = buffer[i];
- int length = _reader.getDictIdMV(nextDocId, _dictIdBuffer, _readerContext);
- _numEntriesScanned += length;
- if (_predicateEvaluator.applyMV(_dictIdBuffer, length)) {
+ int[] length = new int[]{0};
Review Comment:
This seems like it can be avoided. Unnecessary object construction for every iteration in the loop ? Can we avoid passing this to doesValueMatch for pass by reference ? If this is needed to increment , then we can increment the counter in the matcher itself ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910681944
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
+ {
+ // Test a select with filter query on a MV raw column identifier
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertEquals(intVal[0], i % 5);
+ assertEquals(intVal[1], (i % 5) + MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) i % 5);
+ assertEquals(doubleVal[1], (double) (i % 5) + MV_OFFSET);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), i % 5);
+ assertEquals(Integer.parseInt(stringVal[1]), (i % 5) + MV_OFFSET);
+ }
+ }
+ {
+ // Test a select with filter query (OR) on two MV raw column identifiers (int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "OR mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ double[] doubleVal = (double[]) values[1];
+ assertTrue(intVal[0] < 5 || intVal[1] < 5 || doubleVal[0] > 1104.0 || doubleVal[1] > 1104.0);
+ }
+ }
+ {
+ // Test a select with filter query on a long MV raw column identifier
+ String query = "SELECT mvRawLongCol from testTable where mvRawLongCol > 1100 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ long[] longVal = (long[]) values[0];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] > 1100 || longVal[1] > 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on a string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol = '1100' LIMIT 10";
Review Comment:
Didn't notice this in the part 1 PR. May be it is better to use some actual varying width string / alphanumeric in the MV String column just to ensure correctness ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910675440
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
Review Comment:
Curious if this is enough to ensure the correctness of the values in MV columns in the ResultTable ? Should we not assert / check for those columns as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r910695632
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -807,4 +1320,283 @@ private void validateAggregateWithGroupByQueryResults(ResultTable resultTable, D
}
}
}
+
+ @Test
+ public void testAggregateWithGroupByOrderByQueries() {
+ {
+ // Aggregation on int columns with group by order by
+ String query = "SELECT COUNTMV(mvIntCol), COUNTMV(mvRawIntCol), SUMMV(mvIntCol), SUMMV(mvRawIntCol), "
+ + "MINMV(mvIntCol), MINMV(mvRawIntCol), MAXMV(mvIntCol), MAXMV(mvRawIntCol), AVGMV(mvIntCol), "
+ + "AVGMV(mvRawIntCol), mvRawLongCol from testTable GROUP BY mvRawLongCol ORDER BY mvRawLongCol";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvIntCol)", "countmv(mvRawIntCol)", "summv(mvIntCol)", "summv(mvRawIntCol)", "minmv(mvIntCol)",
+ "minmv(mvRawIntCol)", "maxmv(mvIntCol)", "maxmv(mvRawIntCol)", "avgmv(mvIntCol)", "avgmv(mvRawIntCol)",
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.LONG
+ });
+ validateAggregateWithGroupByOrderByQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on long columns with order by (same results as simple aggregation)
+ String query = "SELECT COUNTMV(mvLongCol), COUNTMV(mvRawLongCol), SUMMV(mvLongCol), SUMMV(mvRawLongCol), "
+ + "MINMV(mvLongCol), MINMV(mvRawLongCol), MAXMV(mvLongCol), MAXMV(mvRawLongCol), AVGMV(mvLongCol), "
+ + "AVGMV(mvRawLongCol), mvRawIntCol from testTable GROUP BY mvRawIntCol ORDER BY mvRawIntCol";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvLongCol)", "countmv(mvRawLongCol)", "summv(mvLongCol)", "summv(mvRawLongCol)", "minmv(mvLongCol)",
+ "minmv(mvRawLongCol)", "maxmv(mvLongCol)", "maxmv(mvRawLongCol)", "avgmv(mvLongCol)", "avgmv(mvRawLongCol)",
+ "mvRawIntCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.INT
+ });
+ validateAggregateWithGroupByOrderByQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on float columns with order by (same results as simple aggregation)
+ String query = "SELECT COUNTMV(mvFloatCol), COUNTMV(mvRawFloatCol), SUMMV(mvFloatCol), SUMMV(mvRawFloatCol), "
+ + "MINMV(mvFloatCol), MINMV(mvRawFloatCol), MAXMV(mvFloatCol), MAXMV(mvRawFloatCol), AVGMV(mvFloatCol), "
+ + "AVGMV(mvRawFloatCol), mvRawIntCol from testTable GROUP BY mvRawIntCol ORDER BY mvRawIntCol";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvFloatCol)", "countmv(mvRawFloatCol)", "summv(mvFloatCol)", "summv(mvRawFloatCol)",
+ "minmv(mvFloatCol)", "minmv(mvRawFloatCol)", "maxmv(mvFloatCol)", "maxmv(mvRawFloatCol)",
+ "avgmv(mvFloatCol)", "avgmv(mvRawFloatCol)", "mvRawIntCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.INT
+ });
+ validateAggregateWithGroupByOrderByQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on double columns with order by (same results as simple aggregation)
+ String query = "SELECT COUNTMV(mvDoubleCol), COUNTMV(mvRawDoubleCol), SUMMV(mvDoubleCol), SUMMV(mvRawDoubleCol), "
+ + "MINMV(mvDoubleCol), MINMV(mvRawDoubleCol), MAXMV(mvDoubleCol), MAXMV(mvRawDoubleCol), AVGMV(mvDoubleCol), "
+ + "AVGMV(mvRawDoubleCol), mvRawIntCol from testTable GROUP BY mvRawIntCol ORDER BY mvRawIntCol";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvDoubleCol)", "countmv(mvRawDoubleCol)", "summv(mvDoubleCol)", "summv(mvRawDoubleCol)",
+ "minmv(mvDoubleCol)", "minmv(mvRawDoubleCol)", "maxmv(mvDoubleCol)", "maxmv(mvRawDoubleCol)",
+ "avgmv(mvDoubleCol)", "avgmv(mvRawDoubleCol)", "mvRawIntCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.INT
+ });
+ validateAggregateWithGroupByOrderByQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on string columns with order by (same results as simple aggregation)
+ String query = "SELECT COUNTMV(mvStringCol), COUNTMV(mvRawStringCol), SUMMV(mvStringCol), SUMMV(mvRawStringCol), "
+ + "MINMV(mvStringCol), MINMV(mvRawStringCol), MAXMV(mvStringCol), MAXMV(mvRawStringCol), AVGMV(mvStringCol), "
+ + "AVGMV(mvRawStringCol), mvRawIntCol from testTable GROUP BY mvRawIntCol ORDER BY mvRawIntCol";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvStringCol)", "countmv(mvRawStringCol)", "summv(mvStringCol)", "summv(mvRawStringCol)",
+ "minmv(mvStringCol)", "minmv(mvRawStringCol)", "maxmv(mvStringCol)", "maxmv(mvRawStringCol)",
+ "avgmv(mvStringCol)", "avgmv(mvRawStringCol)", "mvRawIntCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.INT
+ });
+ validateAggregateWithGroupByOrderByQueryResults(resultTable, dataSchema);
+ }
+ {
+ // Aggregation on int columns with group by order by with order by agg
+ String query = "SELECT COUNTMV(mvIntCol), COUNTMV(mvRawIntCol), SUMMV(mvIntCol), SUMMV(mvRawIntCol), "
+ + "MINMV(mvIntCol), MINMV(mvRawIntCol), MAXMV(mvIntCol), MAXMV(mvRawIntCol), AVGMV(mvIntCol), "
+ + "AVGMV(mvRawIntCol), mvRawIntCol from testTable GROUP BY mvRawIntCol ORDER BY mvRawIntCol DESC, "
+ + "SUMMV(mvRawIntCol) DESC";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "countmv(mvIntCol)", "countmv(mvRawIntCol)", "summv(mvIntCol)", "summv(mvRawIntCol)", "minmv(mvIntCol)",
+ "minmv(mvRawIntCol)", "maxmv(mvIntCol)", "maxmv(mvRawIntCol)", "avgmv(mvIntCol)", "avgmv(mvRawIntCol)",
+ "mvRawIntCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.LONG, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.INT
+ });
+ assertNotNull(resultTable);
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ double[] expectedSumValues = new double[]{8472.0, 8464.0, 8456.0, 8448.0, 8440.0, 8432.0, 8424.0, 8416.0, 8408.0,
+ 8400.0};
+ double[] expectedMinValues = new double[]{1009.0, 1008.0, 1007.0, 1006.0, 1005.0, 1004.0, 1003.0, 1002.0, 1001.0,
+ 1000.0};
+ double[] expectedMaxValues = new double[]{1109.0, 1108.0, 1107.0, 1106.0, 1105.0, 1104.0, 1103.0, 1102.0, 1101.0,
+ 1100.0};
+ double[] expectedAvgValues = new double[]{1059.0, 1058.0, 1057.0, 1056.0, 1055.0, 1054.0, 1053.0, 1052.0, 1051.0,
+ 1050.0};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 11);
+ long count = (long) values[0];
+ long countRaw = (long) values[1];
+ assertEquals(count, 8);
+ assertEquals(count, countRaw);
+
+ double sum = (double) values[2];
+ double sumRaw = (double) values[3];
+ assertEquals(sum, sumRaw);
+ assertEquals(sum, expectedSumValues[i]);
+
+ double min = (double) values[4];
+ double minRaw = (double) values[5];
+ assertEquals(min, minRaw);
+ assertEquals(min, expectedMinValues[i]);
+
+ double max = (double) values[6];
+ double maxRaw = (double) values[7];
+ assertEquals(max, maxRaw);
+ assertEquals(max, expectedMaxValues[i]);
+
+ double avg = (double) values[8];
+ double avgRaw = (double) values[9];
+ assertEquals(avg, avgRaw);
+ assertEquals(avg, expectedAvgValues[i]);
+
+ assertEquals((int) values[10], (int) max);
+ }
+ }
+ }
+
+ private void validateAggregateWithGroupByOrderByQueryResults(ResultTable resultTable, DataSchema expectedDataSchema) {
+ assertNotNull(resultTable);
+ assertEquals(resultTable.getDataSchema(), expectedDataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ double[] expectedSumValues = new double[]{400.0, 408.0, 416.0, 424.0, 432.0, 440.0, 448.0, 456.0, 464.0, 472.0};
+ double[] expectedMinValues = new double[]{0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0};
+ double[] expectedMaxValues = new double[]{100.0, 101.0, 102.0, 103.0, 104.0, 105.0, 106.0, 107.0, 108.0, 109.0};
+ double[] expectedAvgValues = new double[]{50.0, 51.0, 52.0, 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 11);
+ long count = (long) values[0];
+ long countRaw = (long) values[1];
+ assertEquals(count, 8);
+ assertEquals(count, countRaw);
+
+ double sum = (double) values[2];
+ double sumRaw = (double) values[3];
+ assertEquals(sum, sumRaw);
+ assertEquals(sum, expectedSumValues[i]);
+
+ double min = (double) values[4];
+ double minRaw = (double) values[5];
+ assertEquals(min, minRaw);
+ assertEquals(min, expectedMinValues[i]);
+
+ double max = (double) values[6];
+ double maxRaw = (double) values[7];
+ assertEquals(max, maxRaw);
+ assertEquals(max, expectedMaxValues[i]);
+
+ double avg = (double) values[8];
+ double avgRaw = (double) values[9];
+ assertEquals(avg, avgRaw);
+ assertEquals(avg, expectedAvgValues[i]);
+
+ if (expectedDataSchema.getColumnDataType(10) == DataSchema.ColumnDataType.LONG) {
+ assertEquals((long) values[10], i);
+ } else {
+ assertEquals((int) values[10], i);
+ }
+ }
+ }
+
+ @Test
+ public void testTransformInsideAggregateQueries() {
Review Comment:
Just saw this. Please ignore my earlier comment on nesting transform inside aggregate. It seems covered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911172129
##########
pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/NoDictionarySingleColumnGroupKeyGenerator.java:
##########
@@ -164,8 +166,113 @@ private Map createGroupKeyMap(DataType keyType) {
@Override
public void generateKeysForBlock(TransformBlock transformBlock, int[][] groupKeys) {
Review Comment:
This function is already implemented for `NoDictionaryMultiColumnGroupKeyGenerator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911179077
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
Review Comment:
yes only a subset of rows have value '0' in the column. in such rows, only one value is '0' and the other value is different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] somandal commented on a diff in pull request #8993: Part 2 - Add support for multi-value noDict column types for group-by and filter type queries
Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #8993:
URL: https://github.com/apache/pinot/pull/8993#discussion_r911267892
##########
pinot-core/src/test/java/org/apache/pinot/queries/MultiValueRawQueriesTest.java:
##########
@@ -338,6 +338,445 @@ public void testSelectQueries() {
}
}
+ @Test
+ public void testNonAggregateMVGroupBY() {
+ {
+ // TODO: Today ORDER BY on MV columns (irrespective of whether it's dictionary based or raw) doesn't work
+ // Fix ORDER BY only for MV columns
+ String query = "SELECT mvFloatCol from testTable WHERE mvFloatCol < 5 ORDER BY mvFloatCol LIMIT 10";
+ BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+ assertEquals(brokerResponseNative.getProcessingExceptions().size(), 2);
+ }
+ {
+ // Test a group by query on some raw MV rows. Order by on SV column added for determinism
+ String query = "SELECT svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY "
+ + "svIntCol, mvRawFloatCol, mvRawDoubleCol, mvRawStringCol ORDER BY svIntCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "svIntCol", "mvRawFloatCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.FLOAT, DataSchema.ColumnDataType.DOUBLE,
+ DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ int[] expectedSVInts = new int[]{0, 0, 0, 0, 0, 0, 0, 0, 1, 1};
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 4);
+ assertEquals((int) values[0], expectedSVInts[i]);
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawIntCol, mvRawDoubleCol LIMIT 20";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 20);
+
+ int intValue = -1;
+ double doubleValue = -1;
+ for (int i = 0; i < 20; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int intValueCur = (int) values[0];
+ double doubleValueCur = (double) values[1];
+ assertTrue(intValueCur >= intValue);
+ if (intValueCur == intValue) {
+ assertTrue(doubleValueCur >= doubleValue);
+ }
+
+ intValue = intValueCur;
+ doubleValue = doubleValueCur;
+ }
+ }
+ {
+ // Test a group by order by query on some raw MV rows (order by on string)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable GROUP BY mvRawIntCol, "
+ + "mvRawDoubleCol, mvRawStringCol ORDER BY mvRawStringCol LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.STRING
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ String stringVal = null;
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ String stringValueCur = (String) values[2];
+ if (stringVal != null && !stringValueCur.equals(stringVal)) {
+ assertTrue(stringValueCur.compareTo(stringVal) > 0);
+ }
+ stringVal = stringValueCur;
+ }
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by
+ String query = "SELECT VALUEIN(mvRawIntCol, '0') from testTable WHERE mvRawIntCol IN (0) GROUP BY "
+ + "VALUEIN(mvRawIntCol, '0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawIntCol,'0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ int intRawVal = (int) values[0];
+ assertEquals(intRawVal, 0);
+ }
+ {
+ // Test a select with a VALUEIN transform function with group by order by
+ String query = "SELECT VALUEIN(mvRawDoubleCol, '0.0') from testTable WHERE mvRawDoubleCol IN (0.0) GROUP BY "
+ + "VALUEIN(mvRawDoubleCol, '0.0') ORDER BY VALUEIN(mvRawDoubleCol, '0.0') LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{"valuein(mvRawDoubleCol,'0.0')"},
+ new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.DOUBLE});
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 1);
+ Object[] values = recordRows.get(0);
+ assertEquals(values.length, 1);
+ double doubleRawVal = (double) values[0];
+ assertEquals(doubleRawVal, 0.0);
+ }
+ }
+
+ @Test
+ public void testSelectWithFilterQueries() {
+ {
+ // Test a select with filter query on a MV raw column identifier
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ assertEquals(intVal[0], i % 5);
+ assertEquals(intVal[1], (i % 5) + MV_OFFSET);
+
+ double[] doubleVal = (double[]) values[1];
+ assertEquals(doubleVal[0], (double) i % 5);
+ assertEquals(doubleVal[1], (double) (i % 5) + MV_OFFSET);
+
+ String[] stringVal = (String[]) values[2];
+ assertEquals(Integer.parseInt(stringVal[0]), i % 5);
+ assertEquals(Integer.parseInt(stringVal[1]), (i % 5) + MV_OFFSET);
+ }
+ }
+ {
+ // Test a select with filter query (OR) on two MV raw column identifiers (int and double)
+ String query = "SELECT mvRawIntCol, mvRawDoubleCol, mvRawStringCol from testTable where mvRawIntCol < 5 "
+ + "OR mvRawDoubleCol > 1104.0 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawIntCol", "mvRawDoubleCol", "mvRawStringCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.INT_ARRAY, DataSchema.ColumnDataType.DOUBLE_ARRAY,
+ DataSchema.ColumnDataType.STRING_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 3);
+ int[] intVal = (int[]) values[0];
+ assertEquals(intVal[1] - intVal[0], MV_OFFSET);
+ double[] doubleVal = (double[]) values[1];
+ assertTrue(intVal[0] < 5 || intVal[1] < 5 || doubleVal[0] > 1104.0 || doubleVal[1] > 1104.0);
+ }
+ }
+ {
+ // Test a select with filter query on a long MV raw column identifier
+ String query = "SELECT mvRawLongCol from testTable where mvRawLongCol > 1100 LIMIT 10";
+ ResultTable resultTable = getBrokerResponse(query).getResultTable();
+ assertNotNull(resultTable);
+ DataSchema dataSchema = new DataSchema(new String[]{
+ "mvRawLongCol"
+ }, new DataSchema.ColumnDataType[]{
+ DataSchema.ColumnDataType.LONG_ARRAY
+ });
+ assertEquals(resultTable.getDataSchema(), dataSchema);
+ List<Object[]> recordRows = resultTable.getRows();
+ assertEquals(recordRows.size(), 10);
+
+ for (int i = 0; i < 10; i++) {
+ Object[] values = recordRows.get(i);
+ assertEquals(values.length, 1);
+ long[] longVal = (long[]) values[0];
+ assertEquals(longVal[1] - longVal[0], MV_OFFSET);
+ assertTrue(longVal[0] > 1100 || longVal[1] > 1100);
+ }
+ }
+ {
+ // Test a select with filter = query on a string MV raw column identifier
+ String query = "SELECT mvRawStringCol from testTable where mvRawStringCol = '1100' LIMIT 10";
Review Comment:
As discussed offline we'll add a future PR in which we have alpha-numeric string MV raw column and add tests for this. Here we do have some varying length string values (len 1, 2, 3, 4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org