You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/07/07 00:26:28 UTC
[GitHub] [druid] clintropolis opened a new pull request, #12753: druid nested data column type
clintropolis opened a new pull request, #12753:
URL: https://github.com/apache/druid/pull/12753
### Description
#12695
_TBD_
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #12753:
URL: https://github.com/apache/druid/pull/12753#discussion_r916101924
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
Review Comment:
Please add the actual type of field in the error message.
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
+ try {
+ parsed = JSON_MAPPER.readValue(arg.asString(), Object.class);
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE("Bad string input [%s] to [%s]", arg.asString(), name());
+ }
+ }
+ return ExprEval.ofComplex(
+ TYPE,
+ parsed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ParseJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ParseJsonExpr(args);
+ }
+ }
+
+
+
+ public static class GetPathExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "get_path";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class GetPathExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public GetPathExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new GetPathExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new GetPathExpr(args);
+ }
+ }
+
+ public static class JsonQueryExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_query";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonQueryExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonQueryExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.ofComplex(
+ TYPE,
+ NestedPathFinder.find(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonQueryExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // call all the output JSON typed
+ return TYPE;
+ }
+ }
+ return new JsonQueryExpr(args);
+ }
+ }
+
+ public static class JsonValueExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_value";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonValueExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonValueExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonValueExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new JsonValueExpr(args);
+ }
+ }
+
+ public static class ListPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class ListPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ ImmutableList.copyOf(info.getLiteralFields())
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListPathsExpr(args);
+ }
+ }
+
+ public static class JsonPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class JsonPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ // maybe in the future ProcessResults should deal in PathFinder.PathPart instead of strings for fields
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ List<String> transformed = info.getLiteralFields()
+ .stream()
+ .map(p -> NestedPathFinder.toNormalizedJsonPath(NestedPathFinder.parseJqPath(p)))
+ .collect(Collectors.toList());
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ transformed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new JsonPathsExpr(args);
+ }
+ }
+
+ public static class ListKeysExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_keys";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class ListKeysExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListKeysExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ NestedPathFinder.findKeys(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListKeysExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListKeysExpr(args);
+ }
+ }
+
+ @Nullable
+ static Object maybeUnwrapStructuredData(ExprEval input)
+ {
+ if (input.value() instanceof StructuredData) {
+ StructuredData data = (StructuredData) input.value();
+ return data.getValue();
+ }
+ return input.value();
+ }
+
+ static void checkArg0NestedType(String fnName, List<Expr> args, ExprEval input)
Review Comment:
nit - you could just pass args.get(0) as an argument. since the intention to use 0th argument is explicit in the function name.
##########
processing/src/main/java/org/apache/druid/segment/NestedDataColumnMerger.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnDescriptor;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.incremental.IncrementalIndex;
+import org.apache.druid.segment.incremental.IncrementalIndexAdapter;
+import org.apache.druid.segment.nested.CompressedNestedDataComplexColumn;
+import org.apache.druid.segment.nested.GlobalDictionarySortedCollector;
+import org.apache.druid.segment.nested.NestedDataColumnSerializer;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedLiteralTypeInfo;
+import org.apache.druid.segment.serde.ComplexColumnPartSerde;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.IntBuffer;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+public class NestedDataColumnMerger implements DimensionMergerV9
+{
+ private static final Logger log = new Logger(NestedDataColumnMerger.class);
+ public static final Comparator<Pair<Integer, PeekingIterator<Long>>> LONG_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+ public static final Comparator<Pair<Integer, PeekingIterator<Double>>> DOUBLE_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+
+ private final String name;
+ private final Closer closer;
+
+ private NestedDataColumnSerializer serializer;
+
+ public NestedDataColumnMerger(
+ String name,
+ IndexSpec indexSpec,
+ SegmentWriteOutMedium segmentWriteOutMedium,
+ ProgressIndicator progressIndicator,
+ Closer closer
+ )
+ {
+
+ this.name = name;
+ this.serializer = new NestedDataColumnSerializer(name, indexSpec, segmentWriteOutMedium, progressIndicator, closer);
+ this.closer = closer;
+ }
+
+ @Override
+ public void writeMergedValueDictionary(List<IndexableAdapter> adapters) throws IOException
+ {
+
+ long dimStartTime = System.currentTimeMillis();
+
+ int numMergeIndex = 0;
+ GlobalDictionarySortedCollector sortedLookup = null;
+ final Indexed[] sortedLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedLongLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedDoubleLookups = new Indexed[adapters.size()];
+
+ final SortedMap<String, NestedLiteralTypeInfo.MutableTypeSet> mergedFields = new TreeMap<>();
+
+ for (int i = 0; i < adapters.size(); i++) {
+ final IndexableAdapter adapter = adapters.get(i);
+ final GlobalDictionarySortedCollector dimValues;
+ if (adapter instanceof IncrementalIndexAdapter) {
+ dimValues = getSortedIndexFromIncrementalAdapter((IncrementalIndexAdapter) adapter, mergedFields);
+ } else if (adapter instanceof QueryableIndexIndexableAdapter) {
+ dimValues = getSortedIndexesFromQueryableAdapter((QueryableIndexIndexableAdapter) adapter, mergedFields);
+ } else {
+ throw new ISE("Unable to merge columns of unsupported adapter %s", adapter.getClass());
+ }
+
+ boolean allNulls = allNull(dimValues.getSortedStrings()) &&
+ allNull(dimValues.getSortedLongs()) &&
+ allNull(dimValues.getSortedDoubles());
+ sortedLookup = dimValues;
Review Comment:
shouldn't this be inside the `if` block below?
##########
processing/src/main/java/org/apache/druid/segment/NestedDataColumnIndexer.java:
##########
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.collections.bitmap.BitmapFactory;
+import org.apache.druid.collections.bitmap.MutableBitmap;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnCapabilitiesImpl;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.data.CloseableIndexed;
+import org.apache.druid.segment.incremental.IncrementalIndex;
+import org.apache.druid.segment.incremental.IncrementalIndexRowHolder;
+import org.apache.druid.segment.nested.GlobalDimensionDictionary;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedLiteralTypeInfo;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.Objects;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+public class NestedDataColumnIndexer implements DimensionIndexer<StructuredData, StructuredData, StructuredData>
Review Comment:
can you add some javadocs here?
##########
processing/src/main/java/org/apache/druid/segment/data/CompressedBlockSerializer.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import org.apache.druid.io.Channels;
+import org.apache.druid.java.util.common.io.smoosh.FileSmoosher;
+import org.apache.druid.segment.CompressedPools;
+import org.apache.druid.segment.serde.MetaSerdeHelper;
+import org.apache.druid.segment.serde.Serializer;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+import org.apache.druid.segment.writeout.WriteOutBytes;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.channels.WritableByteChannel;
+
+public class CompressedBlockSerializer implements Serializer
+{
+ private static final MetaSerdeHelper<CompressedBlockSerializer> META_SERDE_HELPER = MetaSerdeHelper
+ .firstWriteByte((CompressedBlockSerializer x) -> (byte) 0x01)
Review Comment:
nit - can you directly point to the VERSION variable in CompressedBlockReader?
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
Review Comment:
Please add the size of arguments in the exception error message.
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,669 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.NestedPathPart;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
Review Comment:
this one seems a bit odd to me. this method doesn't fail if `maybeJson` returns false. But this method fails, if `maybeJson` returns true and we can't parse the json. but its possible that it was not in fact a json object.
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,658 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.NestedPathPart;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
+ try {
+ parsed = JSON_MAPPER.readValue(arg.asString(), Object.class);
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE("Bad string input [%s] to [%s]", arg.asString(), name());
+ }
+ }
+ return ExprEval.ofComplex(
+ TYPE,
+ parsed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ParseJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ParseJsonExpr(args);
+ }
+ }
+
+
+
+ public static class GetPathExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "get_path";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class GetPathExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public GetPathExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new GetPathExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new GetPathExpr(args);
+ }
+ }
+
+ public static class JsonQueryExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_query";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonQueryExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonQueryExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ NestedPathFinder.find(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonQueryExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // call all the output JSON typed
+ return TYPE;
+ }
+ }
+ return new JsonQueryExpr(args);
+ }
+ }
+
+ public static class JsonValueExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_value";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonValueExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonValueExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonValueExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new JsonValueExpr(args);
+ }
+ }
+
+ public static class ListPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class ListPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ ImmutableList.copyOf(info.getLiteralFields())
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListPathsExpr(args);
+ }
+ }
+
+ public static class JsonPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class JsonPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ // maybe in the future ProcessResults should deal in PathFinder.PathPart instead of strings for fields
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ List<String> transformed = info.getLiteralFields()
+ .stream()
+ .map(p -> NestedPathFinder.toNormalizedJsonPath(NestedPathFinder.parseJqPath(p)))
+ .collect(Collectors.toList());
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ transformed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new JsonPathsExpr(args);
+ }
+ }
+
+ public static class ListKeysExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_keys";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class ListKeysExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListKeysExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ NestedPathFinder.findKeys(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListKeysExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListKeysExpr(args);
+ }
+ }
+
+ @Nullable
+ static Object maybeUnwrapStructuredData(ExprEval input)
+ {
+ if (input.value() instanceof StructuredData) {
+ StructuredData data = (StructuredData) input.value();
+ return data.getValue();
+ }
+ return input.value();
+ }
+
+ static List<NestedPathPart> getArg1PathPartsFromLiteral(String fnName, List<Expr> args)
+ {
+ if (!(args.get(1).isLiteral() && args.get(1).getLiteralValue() instanceof String)) {
+ throw new IAE(
+ "Function[%s] second argument [%s] must be a literal [%s] value",
+ fnName,
+ args.get(1).stringify(),
+ ExpressionType.STRING
+ );
+ }
+ final String path = (String) args.get(1).getLiteralValue();
+ List<NestedPathPart> parts;
+ try {
+ parts = NestedPathFinder.parseJsonPath(path);
+ }
+ catch (IllegalArgumentException iae) {
+ parts = NestedPathFinder.parseJqPath(path);
+ }
+ return parts;
+ }
+
+ static List<NestedPathPart> getArg1JsonPathPartsFromLiteral(String fnName, List<Expr> args)
+ {
+ if (!(args.get(1).isLiteral() && args.get(1).getLiteralValue() instanceof String)) {
+ throw new IAE(
+ "Function[%s] second argument [%s] must be a literal [%s] value",
+ fnName,
+ args.get(1).stringify(),
+ ExpressionType.STRING
+ );
+ }
+ final List<NestedPathPart> parts = NestedPathFinder.parseJsonPath(
+ (String) args.get(1).getLiteralValue()
+ );
+ return parts;
+ }
+
+ static boolean maybeJson(@Nullable String val)
Review Comment:
nit: can be renamed to `maybeIsJson`
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,658 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.NestedPathPart;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
+ try {
+ parsed = JSON_MAPPER.readValue(arg.asString(), Object.class);
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE("Bad string input [%s] to [%s]", arg.asString(), name());
+ }
+ }
+ return ExprEval.ofComplex(
+ TYPE,
+ parsed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ParseJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ParseJsonExpr(args);
+ }
+ }
+
+
+
+ public static class GetPathExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "get_path";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class GetPathExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public GetPathExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new GetPathExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new GetPathExpr(args);
+ }
+ }
+
+ public static class JsonQueryExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_query";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonQueryExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonQueryExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ NestedPathFinder.find(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonQueryExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // call all the output JSON typed
+ return TYPE;
+ }
+ }
+ return new JsonQueryExpr(args);
+ }
+ }
+
+ public static class JsonValueExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_value";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonValueExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonValueExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonValueExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new JsonValueExpr(args);
+ }
+ }
+
+ public static class ListPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class ListPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ ImmutableList.copyOf(info.getLiteralFields())
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListPathsExpr(args);
+ }
+ }
+
+ public static class JsonPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class JsonPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ // maybe in the future ProcessResults should deal in PathFinder.PathPart instead of strings for fields
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ List<String> transformed = info.getLiteralFields()
+ .stream()
+ .map(p -> NestedPathFinder.toNormalizedJsonPath(NestedPathFinder.parseJqPath(p)))
+ .collect(Collectors.toList());
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ transformed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new JsonPathsExpr(args);
+ }
+ }
+
+ public static class ListKeysExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_keys";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class ListKeysExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListKeysExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ NestedPathFinder.findKeys(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListKeysExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListKeysExpr(args);
+ }
+ }
+
+ @Nullable
+ static Object maybeUnwrapStructuredData(ExprEval input)
+ {
+ if (input.value() instanceof StructuredData) {
+ StructuredData data = (StructuredData) input.value();
+ return data.getValue();
+ }
+ return input.value();
+ }
+
+ static List<NestedPathPart> getArg1PathPartsFromLiteral(String fnName, List<Expr> args)
Review Comment:
why not just take args.get(1) as the argument?
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,679 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
+ try {
+ parsed = JSON_MAPPER.readValue(arg.asString(), Object.class);
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE("Bad string input [%s] to [%s]", arg.asString(), name());
+ }
+ }
+ return ExprEval.ofComplex(
+ TYPE,
+ parsed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ParseJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ParseJsonExpr(args);
+ }
+ }
+
+
+
+ public static class GetPathExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "get_path";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class GetPathExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public GetPathExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new GetPathExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new GetPathExpr(args);
+ }
+ }
+
+ public static class JsonQueryExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_query";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonQueryExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonQueryExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.ofComplex(
+ TYPE,
+ NestedPathFinder.find(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonQueryExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // call all the output JSON typed
+ return TYPE;
+ }
+ }
+ return new JsonQueryExpr(args);
+ }
+ }
+
+ public static class JsonValueExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_value";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonValueExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonValueExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonValueExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new JsonValueExpr(args);
+ }
+ }
+
+ public static class ListPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class ListPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ ImmutableList.copyOf(info.getLiteralFields())
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListPathsExpr(args);
+ }
+ }
+
+ public static class JsonPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class JsonPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ // maybe in the future ProcessResults should deal in PathFinder.PathPart instead of strings for fields
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ List<String> transformed = info.getLiteralFields()
+ .stream()
+ .map(p -> NestedPathFinder.toNormalizedJsonPath(NestedPathFinder.parseJqPath(p)))
+ .collect(Collectors.toList());
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ transformed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new JsonPathsExpr(args);
+ }
+ }
+
+ public static class ListKeysExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_keys";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathFinder.NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class ListKeysExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListKeysExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ checkArg0NestedType(name, args, input);
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ NestedPathFinder.findKeys(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListKeysExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListKeysExpr(args);
+ }
+ }
+
+ @Nullable
+ static Object maybeUnwrapStructuredData(ExprEval input)
+ {
+ if (input.value() instanceof StructuredData) {
+ StructuredData data = (StructuredData) input.value();
+ return data.getValue();
+ }
+ return input.value();
+ }
+
+ static void checkArg0NestedType(String fnName, List<Expr> args, ExprEval input)
+ {
+ if (!TYPE.equals(input.type()) && input.value() != null) {
+ // give unknown complex a try, sometimes ingestion needs a little help because input format isn't mapped to a
+ // druid schema
+ if (ExpressionType.UNKNOWN_COMPLEX.equals(input.type())) {
+ return;
+ }
+ throw new IAE(
+ "Function[%s] first argument [%s] must be type [%s] as input, got [%s]",
+ fnName,
+ args.get(0).stringify(),
+ TYPE.asTypeString(),
+ input.type().asTypeString()
+ );
+ }
+ }
+
+ static List<NestedPathFinder.NestedPathPart> getArg1PathPartsFromLiteral(String fnName, List<Expr> args)
Review Comment:
nit. you could just pass the args.get(1) itself as an argument.
##########
processing/src/main/java/org/apache/druid/segment/ComparatorDimensionDictionary.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.segment;
+
+import it.unimi.dsi.fastutil.objects.Object2IntMap;
+import it.unimi.dsi.fastutil.objects.Object2IntOpenHashMap;
+
+import javax.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * {@link Comparator} based {@link DimensionDictionary}
+ *
+ * there are a lot of unused methods in here for now since the only thing this is used for is to build up
+ * the unsorted dictionary and then it is converted to a {@link ComparatorSortedDimensionDictionary}, but
+ * leaving the unused methods in place for now to be basically compatible with the other implementation.
+ *
+ * This version is not thread-safe since the only current user doesn't use the dictionary for realtime queries, if this
+ * changes we need to add a 'ConcurrentComparatorDimensionDictionary'.
+ */
+public class ComparatorDimensionDictionary<T>
+{
+ public static final int ABSENT_VALUE_ID = -1;
+
+ @Nullable
+ private T minValue = null;
+ @Nullable
+ private T maxValue = null;
+ private volatile int idForNull = ABSENT_VALUE_ID;
+
+ private final AtomicLong sizeInBytes = new AtomicLong(0L);
+ private final Object2IntMap<T> valueToId = new Object2IntOpenHashMap<>();
+
+ private final List<T> idToValue = new ArrayList<>();
+ private final Comparator<T> comparator;
+
+ public ComparatorDimensionDictionary(Comparator<T> comparator)
+ {
+ this.comparator = comparator;
+ valueToId.defaultReturnValue(ABSENT_VALUE_ID);
+ }
+
+ public int getId(@Nullable T value)
+ {
+ if (value == null) {
+ return idForNull;
+ }
+ return valueToId.getInt(value);
+ }
+
+ @SuppressWarnings("unused")
+ @Nullable
+ public T getValue(int id)
+ {
+ if (id == idForNull) {
+ return null;
+ }
+ return idToValue.get(id);
+ }
+
+ public int size()
+ {
+ // using idToValue rather than valueToId because the valueToId doesn't account null value, if it is present.
+ return idToValue.size();
+ }
+
+ /**
+ * Gets the current size of this dictionary in bytes.
+ *
+ * @throws IllegalStateException if size computation is disabled.
+ */
+ public long sizeInBytes()
+ {
+ return sizeInBytes.get();
+ }
+
+ public int add(@Nullable T originalValue)
+ {
+ if (originalValue == null) {
+ if (idForNull == ABSENT_VALUE_ID) {
+ idForNull = idToValue.size();
+ idToValue.add(null);
+ }
+ return idForNull;
+ }
+ int prev = valueToId.getInt(originalValue);
+ if (prev >= 0) {
+ return prev;
+ }
+ final int index = idToValue.size();
+ valueToId.put(originalValue, index);
+ idToValue.add(originalValue);
+
+ // Add size of new dim value and 2 references (valueToId and idToValue)
+ sizeInBytes.addAndGet(estimateSizeOfValue(originalValue) + (2L * Long.BYTES));
+
+ minValue = minValue == null || comparator.compare(minValue, originalValue) > 0 ? originalValue : minValue;
+ maxValue = maxValue == null || comparator.compare(maxValue, originalValue) < 0 ? originalValue : maxValue;
+ return index;
+ }
+
+ @SuppressWarnings("unused")
+ public T getMinValue()
+ {
+ return minValue;
+ }
+
+ @SuppressWarnings("unused")
+ public T getMaxValue()
+ {
+ return maxValue;
+ }
+
+ @SuppressWarnings("unused")
+ public int getIdForNull()
+ {
+ return idForNull;
+ }
+
+ public ComparatorSortedDimensionDictionary<T> sort()
+ {
+ return new ComparatorSortedDimensionDictionary<T>(idToValue, comparator, idToValue.size());
+ }
+
+ /**
+ * Estimates the size of the dimension value in bytes. This method is called
+ * only when a new dimension value is being added to the lookup.
+ *
+ * @throws UnsupportedOperationException Implementations that want to estimate
Review Comment:
what is the fallback if they don't want to estimate memory?
##########
processing/src/main/java/org/apache/druid/segment/NestedDataColumnMerger.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnDescriptor;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.incremental.IncrementalIndex;
+import org.apache.druid.segment.incremental.IncrementalIndexAdapter;
+import org.apache.druid.segment.nested.CompressedNestedDataComplexColumn;
+import org.apache.druid.segment.nested.GlobalDictionarySortedCollector;
+import org.apache.druid.segment.nested.NestedDataColumnSerializer;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedLiteralTypeInfo;
+import org.apache.druid.segment.serde.ComplexColumnPartSerde;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.IntBuffer;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+public class NestedDataColumnMerger implements DimensionMergerV9
+{
+ private static final Logger log = new Logger(NestedDataColumnMerger.class);
+ public static final Comparator<Pair<Integer, PeekingIterator<Long>>> LONG_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+ public static final Comparator<Pair<Integer, PeekingIterator<Double>>> DOUBLE_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+
+ private final String name;
+ private final Closer closer;
+
+ private NestedDataColumnSerializer serializer;
+
+ public NestedDataColumnMerger(
+ String name,
+ IndexSpec indexSpec,
+ SegmentWriteOutMedium segmentWriteOutMedium,
+ ProgressIndicator progressIndicator,
+ Closer closer
+ )
+ {
+
+ this.name = name;
+ this.serializer = new NestedDataColumnSerializer(name, indexSpec, segmentWriteOutMedium, progressIndicator, closer);
+ this.closer = closer;
+ }
+
+ @Override
+ public void writeMergedValueDictionary(List<IndexableAdapter> adapters) throws IOException
+ {
+
+ long dimStartTime = System.currentTimeMillis();
+
+ int numMergeIndex = 0;
+ GlobalDictionarySortedCollector sortedLookup = null;
+ final Indexed[] sortedLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedLongLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedDoubleLookups = new Indexed[adapters.size()];
+
+ final SortedMap<String, NestedLiteralTypeInfo.MutableTypeSet> mergedFields = new TreeMap<>();
+
+ for (int i = 0; i < adapters.size(); i++) {
+ final IndexableAdapter adapter = adapters.get(i);
+ final GlobalDictionarySortedCollector dimValues;
+ if (adapter instanceof IncrementalIndexAdapter) {
+ dimValues = getSortedIndexFromIncrementalAdapter((IncrementalIndexAdapter) adapter, mergedFields);
+ } else if (adapter instanceof QueryableIndexIndexableAdapter) {
+ dimValues = getSortedIndexesFromQueryableAdapter((QueryableIndexIndexableAdapter) adapter, mergedFields);
+ } else {
+ throw new ISE("Unable to merge columns of unsupported adapter %s", adapter.getClass());
+ }
+
+ boolean allNulls = allNull(dimValues.getSortedStrings()) &&
+ allNull(dimValues.getSortedLongs()) &&
+ allNull(dimValues.getSortedDoubles());
+ sortedLookup = dimValues;
+ if (!allNulls) {
+ sortedLookups[i] = dimValues.getSortedStrings();
+ sortedLongLookups[i] = dimValues.getSortedLongs();
+ sortedDoubleLookups[i] = dimValues.getSortedDoubles();
+ numMergeIndex++;
+ }
+ }
+
+ serializer.open();
+ serializer.serializeFields(mergedFields);
+
+ int cardinality = 0;
+ if (numMergeIndex > 1) {
+ DictionaryMergingIterator<String> dictionaryMergeIterator = new DictionaryMergingIterator<>(
+ sortedLookups,
+ StringDimensionMergerV9.DICTIONARY_MERGING_COMPARATOR,
+ true
+ );
+ DictionaryMergingIterator<Long> longDictionaryMergeIterator = new DictionaryMergingIterator<>(
+ sortedLongLookups,
+ LONG_MERGING_COMPARATOR,
+ true
+ );
+ DictionaryMergingIterator<Double> doubleDictionaryMergeIterator = new DictionaryMergingIterator<>(
+ sortedDoubleLookups,
+ DOUBLE_MERGING_COMPARATOR,
+ true
+ );
+ serializer.serializeStringDictionary(() -> dictionaryMergeIterator);
+ serializer.serializeLongDictionary(() -> longDictionaryMergeIterator);
+ serializer.serializeDoubleDictionary(() -> doubleDictionaryMergeIterator);
+ cardinality = dictionaryMergeIterator.getCardinality();
+ } else if (numMergeIndex == 1) {
+ serializer.serializeStringDictionary(sortedLookup.getSortedStrings());
+ serializer.serializeLongDictionary(sortedLookup.getSortedLongs());
+ serializer.serializeDoubleDictionary(sortedLookup.getSortedDoubles());
+ cardinality = sortedLookup.size();
+ }
+
+ log.debug(
+ "Completed dim[%s] conversions with cardinality[%,d] in %,d millis.",
+ name,
+ cardinality,
+ System.currentTimeMillis() - dimStartTime
+ );
+ }
+
+ @Nullable
+ private GlobalDictionarySortedCollector getSortedIndexFromIncrementalAdapter(
+ IncrementalIndexAdapter adapter,
+ SortedMap<String, NestedLiteralTypeInfo.MutableTypeSet> mergedFields
+ )
+ {
+ final IncrementalIndex index = adapter.getIncrementalIndex();
+ final IncrementalIndex.DimensionDesc dim = index.getDimension(name);
+ if (dim == null || !(dim.getIndexer() instanceof NestedDataColumnIndexer)) {
+ return null;
+ }
+ final NestedDataColumnIndexer indexer = (NestedDataColumnIndexer) dim.getIndexer();
+ for (Map.Entry<String, NestedDataColumnIndexer.LiteralFieldIndexer> entry : indexer.fieldIndexers.entrySet()) {
+ // skip adding the field if no types are in the set, meaning only null values have been processed
+ if (!entry.getValue().getTypes().isEmpty()) {
+ mergedFields.put(entry.getKey(), entry.getValue().getTypes());
+ }
+ }
+ return indexer.globalDictionary.getSortedCollector();
+ }
+
+ @Nullable
+ private GlobalDictionarySortedCollector getSortedIndexesFromQueryableAdapter(
+ QueryableIndexIndexableAdapter adapter,
+ SortedMap<String, NestedLiteralTypeInfo.MutableTypeSet> mergedFields
+ )
+ {
+ final ColumnHolder columnHolder = adapter.getQueryableIndex().getColumnHolder(name);
+
+ if (columnHolder == null) {
+ return null;
+ }
+
+ final BaseColumn col = columnHolder.getColumn();
+
+ closer.register(col);
Review Comment:
we are doing this two times. another occurrence is inside the function.
##########
processing/src/main/java/org/apache/druid/segment/NestedDataColumnMerger.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnDescriptor;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.incremental.IncrementalIndex;
+import org.apache.druid.segment.incremental.IncrementalIndexAdapter;
+import org.apache.druid.segment.nested.CompressedNestedDataComplexColumn;
+import org.apache.druid.segment.nested.GlobalDictionarySortedCollector;
+import org.apache.druid.segment.nested.NestedDataColumnSerializer;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedLiteralTypeInfo;
+import org.apache.druid.segment.serde.ComplexColumnPartSerde;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.IntBuffer;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+public class NestedDataColumnMerger implements DimensionMergerV9
Review Comment:
some high level javadocs would have been nice.
##########
processing/src/main/java/org/apache/druid/segment/data/CompressedBlockReader.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.collections.ResourceHolder;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.CompressedPools;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.IntBuffer;
+import java.util.function.Supplier;
+
+/**
+ * Reader for a virtual contiguous address range backed by compressed blocks of data.
+ *
+ * Format:
+ * | version (byte) | compression (byte) | num blocks (int) | block size (int) | end offsets | compressed data |
+ *
+ * This mechanism supports two modes of use, the first where callers may ask for a range of data from the underlying
+ * blocks, provided by {@link #getRange(long, int)}. The {@link ByteBuffer} provided by this method may or may not
+ * be valid after additional calls to {@link #getRange(long, int)} or calls to {@link #seekBlock(int)}.
+ *
+ * For fixed width values which are aligned with the block size, callers may also use the method
+ * {@link #getDecompressedDataBuffer()} to have direct access to the current uncompressed block, and use the methods
+ * {@link #loadBlock(long)} to load the correct block and translate a virtual offset into the relative offset, or
+ * {@link #seekBlock(int)} to change which block is currently loaded.
+ *
+ * {@link #getRange(long, int)} uses these same mechanisms internally to supply data.
+ */
+public final class CompressedBlockReader implements Closeable
+{
+ private static final ByteBuffer NULL_VALUE = ByteBuffer.wrap(new byte[0]);
+ public static final byte VERSION = 0x01;
+
+ public static Supplier<CompressedBlockReader> fromByteBuffer(ByteBuffer buffer, ByteOrder byteOrder)
+ {
+ byte versionFromBuffer = buffer.get();
+
+ if (versionFromBuffer == VERSION) {
+ final CompressionStrategy compression = CompressionStrategy.forId(buffer.get());
+ final int blockSize = buffer.getInt();
+ assert CompressedPools.BUFFER_SIZE == blockSize;
+ Preconditions.checkState(
+ blockSize <= CompressedPools.BUFFER_SIZE,
+ "Maximum block size must be less than " + CompressedPools.BUFFER_SIZE
+ );
+ final int numBlocks = buffer.getInt();
+ final int offsetsSize = numBlocks * Integer.BYTES;
+ // buffer is at start of ending offsets
+ final ByteBuffer offsets = buffer.asReadOnlyBuffer().order(byteOrder);
+ offsets.limit(offsets.position() + offsetsSize);
+ final IntBuffer offsetView = offsets.slice().order(byteOrder).asIntBuffer();
+ final int compressedSize = offsetView.get(numBlocks - 1);
+
+ // move to start of compressed data
+ buffer.position(buffer.position() + offsetsSize);
+ final ByteBuffer compressedData = buffer.asReadOnlyBuffer().order(byteOrder);
+ compressedData.limit(compressedData.position() + compressedSize);
+ buffer.position(buffer.position() + compressedSize);
+
+ final ByteBuffer compressedDataView = compressedData.slice().order(byteOrder);
+ return () -> new CompressedBlockReader(
+ compression,
+ numBlocks,
+ blockSize,
+ offsetView.asReadOnlyBuffer(),
+ compressedDataView.asReadOnlyBuffer().order(byteOrder),
+ byteOrder
+ );
+ }
+ throw new IAE("Unknown version[%s]", versionFromBuffer);
+ }
+
+ private final CompressionStrategy.Decompressor decompressor;
+
+ private final int numBlocks;
+ private final int div;
+ private final int rem;
+ private final IntBuffer endOffsetsBuffer;
+ private final ByteBuffer compressedDataBuffer;
+
+ private final ResourceHolder<ByteBuffer> decompressedDataBufferHolder;
+ private final ByteBuffer decompressedDataBuffer;
+
+ private final ByteOrder byteOrder;
+ private final Closer closer;
+ private int currentBlockNumber = -1;
+
+ public CompressedBlockReader(
+ CompressionStrategy compressionStrategy,
+ int numBlocks,
+ int blockSize,
+ IntBuffer endOffsetsBuffer,
+ ByteBuffer compressedDataBuffer,
+ ByteOrder byteOrder
+ )
+ {
+ this.decompressor = compressionStrategy.getDecompressor();
+ this.numBlocks = numBlocks;
+ this.div = Integer.numberOfTrailingZeros(blockSize);
+ this.rem = blockSize - 1;
+ this.endOffsetsBuffer = endOffsetsBuffer;
+ this.compressedDataBuffer = compressedDataBuffer;
+ this.closer = Closer.create();
+ this.decompressedDataBufferHolder = CompressedPools.getByteBuf(byteOrder);
+ closer.register(decompressedDataBufferHolder);
+ this.decompressedDataBuffer = decompressedDataBufferHolder.get();
+ this.decompressedDataBuffer.clear();
+ this.byteOrder = byteOrder;
+ }
+
+ /**
+ * Get size in bytes of virtual contiguous buffer
+ */
+ public long getSize()
Review Comment:
this is compressedSize, correct?
##########
processing/src/main/java/org/apache/druid/segment/data/CompressedBlockReader.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.collections.ResourceHolder;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.CompressedPools;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.IntBuffer;
+import java.util.function.Supplier;
+
+/**
+ * Reader for a virtual contiguous address range backed by compressed blocks of data.
+ *
+ * Format:
+ * | version (byte) | compression (byte) | num blocks (int) | block size (int) | end offsets | compressed data |
Review Comment:
block size comes before number of blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on PR #12753:
URL: https://github.com/apache/druid/pull/12753#issuecomment-1184799105
This is amazing stuff. I am excited to follow this up with a web console PR to support some of this stuff there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12753:
URL: https://github.com/apache/druid/pull/12753#discussion_r932908504
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,669 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.NestedPathPart;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
Review Comment:
hmm yeah, good point, will adjust in a future PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky merged pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
vogievetsky merged PR #12753:
URL: https://github.com/apache/druid/pull/12753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] lgtm-com[bot] commented on pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #12753:
URL: https://github.com/apache/druid/pull/12753#issuecomment-1176947928
This pull request **introduces 5 alerts** when merging 3e7b1c632551b56e725cd30c897a85cf623c2426 into 059aba781a59652c09ddea29b7bd6c7d592fb3ce - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-faf5774c087538cdcd5a17c8c77c20d1e5f80fd2)
**new alerts:**
* 2 for Missing format argument
* 2 for Boxed variable is never null
* 1 for Dereferenced variable may be null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
gianm commented on PR #12753:
URL: https://github.com/apache/druid/pull/12753#issuecomment-1181905372
Nice work! Very cool. I just noticed this when you posted it on Slack. The first thing that comes to mind is questions about the intended maturity of this feature after initial contribution:
- Is it intentionally undocumented in this PR? Do you plan to add documentation?
- Are there any impediments to maintaining forwards compatibility, such that new versions of Druid will always be able to read JSON columns written by older versions? Do you foresee any reason we might want to break compatibility?
- Would you recommend we present this feature in its current state as experimental or production-ready, & why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12753:
URL: https://github.com/apache/druid/pull/12753#discussion_r932908980
##########
processing/src/main/java/org/apache/druid/segment/data/CompressedBlockReader.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.collections.ResourceHolder;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.CompressedPools;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.IntBuffer;
+import java.util.function.Supplier;
+
+/**
+ * Reader for a virtual contiguous address range backed by compressed blocks of data.
+ *
+ * Format:
+ * | version (byte) | compression (byte) | num blocks (int) | block size (int) | end offsets | compressed data |
Review Comment:
hah oops, good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12753:
URL: https://github.com/apache/druid/pull/12753#discussion_r932917683
##########
processing/src/main/java/org/apache/druid/segment/NestedDataColumnMerger.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnDescriptor;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.incremental.IncrementalIndex;
+import org.apache.druid.segment.incremental.IncrementalIndexAdapter;
+import org.apache.druid.segment.nested.CompressedNestedDataComplexColumn;
+import org.apache.druid.segment.nested.GlobalDictionarySortedCollector;
+import org.apache.druid.segment.nested.NestedDataColumnSerializer;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedLiteralTypeInfo;
+import org.apache.druid.segment.serde.ComplexColumnPartSerde;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.IntBuffer;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+public class NestedDataColumnMerger implements DimensionMergerV9
+{
+ private static final Logger log = new Logger(NestedDataColumnMerger.class);
+ public static final Comparator<Pair<Integer, PeekingIterator<Long>>> LONG_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+ public static final Comparator<Pair<Integer, PeekingIterator<Double>>> DOUBLE_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+
+ private final String name;
+ private final Closer closer;
+
+ private NestedDataColumnSerializer serializer;
+
+ public NestedDataColumnMerger(
+ String name,
+ IndexSpec indexSpec,
+ SegmentWriteOutMedium segmentWriteOutMedium,
+ ProgressIndicator progressIndicator,
+ Closer closer
+ )
+ {
+
+ this.name = name;
+ this.serializer = new NestedDataColumnSerializer(name, indexSpec, segmentWriteOutMedium, progressIndicator, closer);
+ this.closer = closer;
+ }
+
+ @Override
+ public void writeMergedValueDictionary(List<IndexableAdapter> adapters) throws IOException
+ {
+
+ long dimStartTime = System.currentTimeMillis();
+
+ int numMergeIndex = 0;
+ GlobalDictionarySortedCollector sortedLookup = null;
+ final Indexed[] sortedLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedLongLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedDoubleLookups = new Indexed[adapters.size()];
+
+ final SortedMap<String, NestedLiteralTypeInfo.MutableTypeSet> mergedFields = new TreeMap<>();
+
+ for (int i = 0; i < adapters.size(); i++) {
+ final IndexableAdapter adapter = adapters.get(i);
+ final GlobalDictionarySortedCollector dimValues;
+ if (adapter instanceof IncrementalIndexAdapter) {
+ dimValues = getSortedIndexFromIncrementalAdapter((IncrementalIndexAdapter) adapter, mergedFields);
+ } else if (adapter instanceof QueryableIndexIndexableAdapter) {
+ dimValues = getSortedIndexesFromQueryableAdapter((QueryableIndexIndexableAdapter) adapter, mergedFields);
+ } else {
+ throw new ISE("Unable to merge columns of unsupported adapter %s", adapter.getClass());
+ }
+
+ boolean allNulls = allNull(dimValues.getSortedStrings()) &&
+ allNull(dimValues.getSortedLongs()) &&
+ allNull(dimValues.getSortedDoubles());
+ sortedLookup = dimValues;
Review Comment:
nah, this is a strange flow that i picked up from this code being based on `DictionaryEncodedColumnMerger`. Every iteration sets sortedLookup, which is only used if there is a single thing being "merged", which _really_ means that an incremental index is being persisted. If there are more than 1 set things being merged, then it uses a `DictionaryMergingIterator` for each locally sorted set of values to merge them, which null values in the array are just skipped.
Its probably worth adjusting this code, will try to re-arrange in the future so its a bit easier to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on PR #12753:
URL: https://github.com/apache/druid/pull/12753#issuecomment-1182535667
>Is it intentionally undocumented in this PR? Do you plan to add documentation?
I was planning to add documentation in a follow-up PR since I thought this one was already big enough 😅
>Are there any impediments to maintaining forwards compatibility of the storage format, such that new versions of Druid will always be able to read JSON columns written by older versions? Do you foresee any reason we might want to break compatibility?
I modeled the column after existing Druid columns so most things are decorated with a version byte which should allow us to make changes in the future while still being able to continue reading the existing data. For the specific list of what is versioned:
* `NestedDataColumnSerializer` for the complex column itself (currently on v3 actually, i removed the reader code for older versions from prototyping to get rid of dead code)
* `GlobalDictionaryEncodedFieldColumnWriter` which writes the nested columns and is currently re-using `DictionaryEncodedColumnPartSerde.VERSION` (i should probably decouple this at some point in the future...)
* `FixedIndexed` (building block used to store local to global dictionary mapping and long and double value dictionaries)
* `CompressedVariableSizedBlobColumnSerializer` (used to compress raw data)
* `CompressedBlockSerializer` (used internally by `CompressedVariableSizedBlobColumnSerializer`)
In the "Future work" section of #12695 I mention storage format as an area that we can iterate on in the future, the biggest things I have in mind right now are storing arrays of literal values as array typed columns instead of broken out as they currently are, as well as customization such as allowing skipping building indexes on certain columns or storing them all-together also probably falls into this. Nothing about the current code should block this afaik, nor should those future enhancements interfere with our ability to read data that is stored with the current versions of stuff, so long as we practice good version hygiene whenever we make changes.
> Would you recommend we present this feature in its current state as experimental or production-ready, & why?
This is a hard one to answer, though I am hesitant to call it production ready right from the start, I think the answer might vary a bit per use case.
The surface area here is huge since it essentially provides all of the normal Druid column functionality within these `COMPLEX<json>` columns, and I definitely won't claim this to be bug free. That said, quite a lot of internal testing has been done at this point, even at scale and with complicated nested schemas, which has allowed this codebase to be iterated on to get it to the place it currently is. There are some rough spots which I'm looking to improve in the near future, such as ingest time memory footprint, better array handling, etc, but I think if we get the documentation in a good enough state and can list out the limitations it could be used today.
The use cases I would feel most comfortable with are replacements for what can currently be done via flattening, meaning not heavily centered on nested arrays. I do have ideas of how to better support nested arrays and my goal is to allow arrays extracted from nested columns to be exposed as druid `ARRAY` types, but I am not there yet, so I'm not sure I would recommend most array use cases unless they are more like vectors which have expected lengths and array positions are known/meaningful (and such that most queries would be extracting specific array positions, not entire arrays).
There is also the matter of different performance characteristics at both ingest and query time for these columns. Ingestion time segment merge is pretty heavy right now because the global value dictionary is stored in heap. Query performance can vary a fair bit with nested columns compared to flat columns, especially with numbers due to the existence of indexes on these numeric columns, which currently at least sometimes results in dramatically faster but also sometimes slower query performance. I'm still exploring this quite a bit, besides documentation follow-up I also have been working on doing some benchmarking to see where things currently stand and plan on sharing those results relatively soon.
So, long story short, due to the unknowns I think the answer for right now is that operators should experiment with `COMPLEX<json>` columns to see if they work well for their use case, and use them in production if so, otherwise provide feedback so that we can continue to make improvements and expand the use cases this is good for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12753:
URL: https://github.com/apache/druid/pull/12753#discussion_r932917683
##########
processing/src/main/java/org/apache/druid/segment/NestedDataColumnMerger.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnDescriptor;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.incremental.IncrementalIndex;
+import org.apache.druid.segment.incremental.IncrementalIndexAdapter;
+import org.apache.druid.segment.nested.CompressedNestedDataComplexColumn;
+import org.apache.druid.segment.nested.GlobalDictionarySortedCollector;
+import org.apache.druid.segment.nested.NestedDataColumnSerializer;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedLiteralTypeInfo;
+import org.apache.druid.segment.serde.ComplexColumnPartSerde;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.IntBuffer;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+public class NestedDataColumnMerger implements DimensionMergerV9
+{
+ private static final Logger log = new Logger(NestedDataColumnMerger.class);
+ public static final Comparator<Pair<Integer, PeekingIterator<Long>>> LONG_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+ public static final Comparator<Pair<Integer, PeekingIterator<Double>>> DOUBLE_MERGING_COMPARATOR =
+ DictionaryMergingIterator.makePeekingComparator();
+
+ private final String name;
+ private final Closer closer;
+
+ private NestedDataColumnSerializer serializer;
+
+ public NestedDataColumnMerger(
+ String name,
+ IndexSpec indexSpec,
+ SegmentWriteOutMedium segmentWriteOutMedium,
+ ProgressIndicator progressIndicator,
+ Closer closer
+ )
+ {
+
+ this.name = name;
+ this.serializer = new NestedDataColumnSerializer(name, indexSpec, segmentWriteOutMedium, progressIndicator, closer);
+ this.closer = closer;
+ }
+
+ @Override
+ public void writeMergedValueDictionary(List<IndexableAdapter> adapters) throws IOException
+ {
+
+ long dimStartTime = System.currentTimeMillis();
+
+ int numMergeIndex = 0;
+ GlobalDictionarySortedCollector sortedLookup = null;
+ final Indexed[] sortedLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedLongLookups = new Indexed[adapters.size()];
+ final Indexed[] sortedDoubleLookups = new Indexed[adapters.size()];
+
+ final SortedMap<String, NestedLiteralTypeInfo.MutableTypeSet> mergedFields = new TreeMap<>();
+
+ for (int i = 0; i < adapters.size(); i++) {
+ final IndexableAdapter adapter = adapters.get(i);
+ final GlobalDictionarySortedCollector dimValues;
+ if (adapter instanceof IncrementalIndexAdapter) {
+ dimValues = getSortedIndexFromIncrementalAdapter((IncrementalIndexAdapter) adapter, mergedFields);
+ } else if (adapter instanceof QueryableIndexIndexableAdapter) {
+ dimValues = getSortedIndexesFromQueryableAdapter((QueryableIndexIndexableAdapter) adapter, mergedFields);
+ } else {
+ throw new ISE("Unable to merge columns of unsupported adapter %s", adapter.getClass());
+ }
+
+ boolean allNulls = allNull(dimValues.getSortedStrings()) &&
+ allNull(dimValues.getSortedLongs()) &&
+ allNull(dimValues.getSortedDoubles());
+ sortedLookup = dimValues;
Review Comment:
nah, this is a strange flow that i picked up from this code being based on `DictionaryEncodedColumnMerger`. Every iteration sets sortedLookup, which is only used if there is a single thing being "merged", which _really_ means that an incremental index is being persisted and so just passes through its sorted dictionary it built. If there are more than 1 set things being merged, then it uses a `DictionaryMergingIterator` to split apart each segments sorted global dictionary and build arrays for each locally sorted set of values to merge them, which null values are just skipped.
Its probably worth adjusting this code, will try to re-arrange in the future so its a bit easier to follow, especially since unlike `DictionaryEncodedColumnMerger` the "indexable adapter" abstraction sort of falls apart and i have to know when i'm dealing with incremental indexes vs queryable indexes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #12753:
URL: https://github.com/apache/druid/pull/12753#discussion_r945523605
##########
processing/src/main/java/org/apache/druid/segment/data/CompressedBlockSerializer.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import org.apache.druid.io.Channels;
+import org.apache.druid.java.util.common.io.smoosh.FileSmoosher;
+import org.apache.druid.segment.CompressedPools;
+import org.apache.druid.segment.serde.MetaSerdeHelper;
+import org.apache.druid.segment.serde.Serializer;
+import org.apache.druid.segment.writeout.SegmentWriteOutMedium;
+import org.apache.druid.segment.writeout.WriteOutBytes;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.channels.WritableByteChannel;
+
+public class CompressedBlockSerializer implements Serializer
+{
+ private static final MetaSerdeHelper<CompressedBlockSerializer> META_SERDE_HELPER = MetaSerdeHelper
+ .firstWriteByte((CompressedBlockSerializer x) -> (byte) 0x01)
+ .writeByte(x -> x.compression.getId())
+ .writeInt(x -> CompressedPools.BUFFER_SIZE)
+ .writeInt(x -> x.numBlocks);
+
+ private final SegmentWriteOutMedium segmentWriteOutMedium;
+ private final CompressionStrategy compression;
+ private final CompressionStrategy.Compressor compressor;
+
+ private final ByteBuffer offsetValueConverter = ByteBuffer.allocate(Integer.BYTES).order(ByteOrder.nativeOrder());
+
+ @Nullable
+ private ByteBuffer uncompressedDataBuffer;
+ private ByteBuffer compressedDataBuffer;
+ private int numBlocks;
+ private int currentOffset;
+
+ @Nullable
+ private WriteOutBytes headerOut = null;
+ @Nullable
+ private WriteOutBytes valuesOut = null;
+
+ public CompressedBlockSerializer(
+ SegmentWriteOutMedium segmentWriteOutMedium,
+
+ CompressionStrategy compression,
+ int blockSize
+ )
+ {
+ this.segmentWriteOutMedium = segmentWriteOutMedium;
+ this.compression = compression;
+ this.compressor = compression.getCompressor();
+ this.uncompressedDataBuffer = compressor.allocateInBuffer(blockSize, segmentWriteOutMedium.getCloser())
+ .order(ByteOrder.nativeOrder());
+ this.compressedDataBuffer = compressor.allocateOutBuffer(blockSize, segmentWriteOutMedium.getCloser())
+ .order(ByteOrder.nativeOrder());
+ }
+
+ public void open() throws IOException
+ {
+ headerOut = segmentWriteOutMedium.makeWriteOutBytes();
+ valuesOut = segmentWriteOutMedium.makeWriteOutBytes();
+ }
+
+ public void addValue(byte[] bytes) throws IOException
+ {
+ if (uncompressedDataBuffer == null) {
+ throw new IllegalStateException("written out already");
+ }
+ flushIfNeeded();
+
+ if (bytes.length <= uncompressedDataBuffer.remaining()) {
+ uncompressedDataBuffer.put(bytes);
+ } else {
+ int written = 0;
+ // write until we have had our fill, flushing buffers as needed
+ while (written < bytes.length) {
+ int next = Math.min(uncompressedDataBuffer.remaining(), bytes.length - written);
+ uncompressedDataBuffer.put(bytes, written, next);
+ written += next;
+ flushIfNeeded();
+ }
+ }
+ }
+
+ public void addValue(ByteBuffer bytes) throws IOException
+ {
+ if (uncompressedDataBuffer == null) {
+ throw new IllegalStateException("written out already");
+ }
+ flushIfNeeded();
+ int size = bytes.remaining();
+ if (size <= uncompressedDataBuffer.remaining()) {
+ uncompressedDataBuffer.put(bytes);
+ } else {
+ ByteBuffer transferBuffer = bytes.asReadOnlyBuffer().order(bytes.order());
+ while (transferBuffer.hasRemaining()) {
+ int writeSize = Math.min(transferBuffer.remaining(), uncompressedDataBuffer.remaining());
+ transferBuffer.limit(transferBuffer.position() + writeSize);
+ uncompressedDataBuffer.put(transferBuffer);
+ transferBuffer.limit(bytes.limit());
+ flushIfNeeded();
+ }
+ }
+ bytes.rewind();
+ }
+
+ @Override
+ public long getSerializedSize() throws IOException
+ {
+ writeEndBuffer();
+ return META_SERDE_HELPER.size(this) + headerOut.size() + valuesOut.size();
+ }
+
+ @Override
+ public void writeTo(WritableByteChannel channel, FileSmoosher smoosher) throws IOException
+ {
+ writeEndBuffer();
+ META_SERDE_HELPER.writeTo(channel, this);
+ headerOut.writeTo(channel);
+ valuesOut.writeTo(channel);
+ }
+
+ private void flushIfNeeded() throws IOException
+ {
+ if (!uncompressedDataBuffer.hasRemaining()) {
+ flushBuffer();
+ }
+ }
+
+ private void flushBuffer() throws IOException
+ {
+ uncompressedDataBuffer.rewind();
+ compressedDataBuffer.clear();
+
+ final ByteBuffer compressed = compressor.compress(uncompressedDataBuffer, compressedDataBuffer);
+
+ currentOffset += compressed.remaining();
+ offsetValueConverter.clear();
+ offsetValueConverter.putInt(currentOffset);
+ offsetValueConverter.flip();
+ Channels.writeFully(headerOut, offsetValueConverter);
+ Channels.writeFully(valuesOut, compressed);
+ uncompressedDataBuffer.clear();
+ numBlocks++;
+ if (numBlocks < 0) {
Review Comment:
how does this happen?
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,669 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.NestedPathPart;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
+ try {
+ parsed = JSON_MAPPER.readValue(arg.asString(), Object.class);
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE("Bad string input [%s] to [%s]", arg.asString(), name());
+ }
+ }
+ return ExprEval.ofComplex(
+ TYPE,
+ parsed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ParseJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ParseJsonExpr(args);
+ }
+ }
+
+
+
+ public static class GetPathExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "get_path";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class GetPathExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public GetPathExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new GetPathExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new GetPathExpr(args);
+ }
+ }
+
+ public static class JsonQueryExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_query";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonQueryExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonQueryExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ NestedPathFinder.find(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonQueryExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // call all the output JSON typed
+ return TYPE;
+ }
+ }
+ return new JsonQueryExpr(args);
+ }
+ }
+
+ public static class JsonValueExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_value";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonValueExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonValueExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonValueExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new JsonValueExpr(args);
+ }
+ }
+
+ public static class ListPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
Review Comment:
this method needs javadoc. what is the significance of return value here?
##########
processing/src/main/java/org/apache/druid/segment/data/CompressedBlockReader.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.collections.ResourceHolder;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.CompressedPools;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.IntBuffer;
+import java.util.function.Supplier;
+
+/**
+ * Reader for a virtual contiguous address range backed by compressed blocks of data.
+ *
+ * Format:
+ * | version (byte) | compression (byte) | num blocks (int) | block size (int) | end offsets | compressed data |
+ *
+ * This mechanism supports two modes of use, the first where callers may ask for a range of data from the underlying
+ * blocks, provided by {@link #getRange(long, int)}. The {@link ByteBuffer} provided by this method may or may not
+ * be valid after additional calls to {@link #getRange(long, int)} or calls to {@link #seekBlock(int)}.
+ *
+ * For fixed width values which are aligned with the block size, callers may also use the method
+ * {@link #getDecompressedDataBuffer()} to have direct access to the current uncompressed block, and use the methods
+ * {@link #loadBlock(long)} to load the correct block and translate a virtual offset into the relative offset, or
+ * {@link #seekBlock(int)} to change which block is currently loaded.
+ *
+ * {@link #getRange(long, int)} uses these same mechanisms internally to supply data.
+ */
+public final class CompressedBlockReader implements Closeable
+{
+ private static final ByteBuffer NULL_VALUE = ByteBuffer.wrap(new byte[0]);
+ public static final byte VERSION = 0x01;
+
+ public static Supplier<CompressedBlockReader> fromByteBuffer(ByteBuffer buffer, ByteOrder byteOrder)
+ {
+ byte versionFromBuffer = buffer.get();
+
+ if (versionFromBuffer == VERSION) {
+ final CompressionStrategy compression = CompressionStrategy.forId(buffer.get());
+ final int blockSize = buffer.getInt();
+ assert CompressedPools.BUFFER_SIZE == blockSize;
+ Preconditions.checkState(
+ blockSize <= CompressedPools.BUFFER_SIZE,
+ "Maximum block size must be less than " + CompressedPools.BUFFER_SIZE
+ );
+ final int numBlocks = buffer.getInt();
+ final int offsetsSize = numBlocks * Integer.BYTES;
+ // buffer is at start of ending offsets
+ final ByteBuffer offsets = buffer.asReadOnlyBuffer().order(byteOrder);
+ offsets.limit(offsets.position() + offsetsSize);
+ final IntBuffer offsetView = offsets.slice().order(byteOrder).asIntBuffer();
+ final int compressedSize = offsetView.get(numBlocks - 1);
+
+ // move to start of compressed data
+ buffer.position(buffer.position() + offsetsSize);
+ final ByteBuffer compressedData = buffer.asReadOnlyBuffer().order(byteOrder);
+ compressedData.limit(compressedData.position() + compressedSize);
+ buffer.position(buffer.position() + compressedSize);
+
+ final ByteBuffer compressedDataView = compressedData.slice().order(byteOrder);
+ return () -> new CompressedBlockReader(
+ compression,
+ numBlocks,
+ blockSize,
+ offsetView.asReadOnlyBuffer(),
+ compressedDataView.asReadOnlyBuffer().order(byteOrder),
+ byteOrder
+ );
+ }
+ throw new IAE("Unknown version[%s]", versionFromBuffer);
+ }
+
+ private final CompressionStrategy.Decompressor decompressor;
+
+ private final int numBlocks;
+ private final int div;
+ private final int rem;
+ private final IntBuffer endOffsetsBuffer;
+ private final ByteBuffer compressedDataBuffer;
+
+ private final ResourceHolder<ByteBuffer> decompressedDataBufferHolder;
+ private final ByteBuffer decompressedDataBuffer;
+
+ private final ByteOrder byteOrder;
+ private final Closer closer;
+ private int currentBlockNumber = -1;
+
+ public CompressedBlockReader(
+ CompressionStrategy compressionStrategy,
+ int numBlocks,
+ int blockSize,
+ IntBuffer endOffsetsBuffer,
+ ByteBuffer compressedDataBuffer,
+ ByteOrder byteOrder
+ )
+ {
+ this.decompressor = compressionStrategy.getDecompressor();
+ this.numBlocks = numBlocks;
+ this.div = Integer.numberOfTrailingZeros(blockSize);
+ this.rem = blockSize - 1;
+ this.endOffsetsBuffer = endOffsetsBuffer;
+ this.compressedDataBuffer = compressedDataBuffer;
+ this.closer = Closer.create();
+ this.decompressedDataBufferHolder = CompressedPools.getByteBuf(byteOrder);
+ closer.register(decompressedDataBufferHolder);
+ this.decompressedDataBuffer = decompressedDataBufferHolder.get();
+ this.decompressedDataBuffer.clear();
+ this.byteOrder = byteOrder;
+ }
+
+ /**
+ * Get size in bytes of virtual contiguous buffer
+ */
+ public long getSize()
Review Comment:
that will be of help. Since At line #75, it says `final int compressedSize = offsetView.get(numBlocks - 1);`
##########
processing/src/main/java/org/apache/druid/query/expression/NestedDataExpressions.java:
##########
@@ -0,0 +1,669 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+package org.apache.druid.query.expression;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.math.expr.Expr;
+import org.apache.druid.math.expr.ExprEval;
+import org.apache.druid.math.expr.ExprMacroTable;
+import org.apache.druid.math.expr.ExprType;
+import org.apache.druid.math.expr.ExpressionType;
+import org.apache.druid.segment.nested.NestedDataComplexTypeSerde;
+import org.apache.druid.segment.nested.NestedPathFinder;
+import org.apache.druid.segment.nested.NestedPathPart;
+import org.apache.druid.segment.nested.StructuredData;
+import org.apache.druid.segment.nested.StructuredDataProcessor;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class NestedDataExpressions
+{
+ private static final ObjectMapper JSON_MAPPER = new DefaultObjectMapper();
+
+ public static final ExpressionType TYPE = Preconditions.checkNotNull(
+ ExpressionType.fromColumnType(NestedDataComplexTypeSerde.TYPE)
+ );
+
+ public static class StructExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "struct";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ Preconditions.checkArgument(args.size() % 2 == 0);
+ class StructExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public StructExpr(List<Expr> args)
+ {
+ super(NAME, args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ HashMap<String, Object> theMap = new HashMap<>();
+ for (int i = 0; i < args.size(); i += 2) {
+ ExprEval field = args.get(i).eval(bindings);
+ ExprEval value = args.get(i + 1).eval(bindings);
+
+ Preconditions.checkArgument(field.type().is(ExprType.STRING), "field name must be a STRING");
+ theMap.put(field.asString(), value.value());
+ }
+
+ return ExprEval.ofComplex(TYPE, theMap);
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new StructExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new StructExpr(args);
+ }
+ }
+
+ public static class JsonObjectExprMacro extends StructExprMacro
+ {
+ public static final String NAME = "json_object";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+ }
+
+ public static class ToJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ maybeUnwrapStructuredData(input)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ToJsonExpr(args);
+ }
+ }
+
+ public static class ToJsonStringExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "to_json_string";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ToJsonStringExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ToJsonStringExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ try {
+ final Object unwrapped = maybeUnwrapStructuredData(input);
+ final String stringify = unwrapped == null ? null : JSON_MAPPER.writeValueAsString(unwrapped);
+ return ExprEval.ofType(
+ ExpressionType.STRING,
+ stringify
+ );
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE(e, "Unable to stringify [%s] to JSON", input.value());
+ }
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ToJsonStringExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING;
+ }
+ }
+ return new ToJsonStringExpr(args);
+ }
+ }
+
+ public static class ParseJsonExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "parse_json";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ class ParseJsonExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ParseJsonExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval arg = args.get(0).eval(bindings);
+ Object parsed = maybeUnwrapStructuredData(arg);
+ if (arg.type().is(ExprType.STRING) && arg.value() != null && maybeJson(arg.asString())) {
+ try {
+ parsed = JSON_MAPPER.readValue(arg.asString(), Object.class);
+ }
+ catch (JsonProcessingException e) {
+ throw new IAE("Bad string input [%s] to [%s]", arg.asString(), name());
+ }
+ }
+ return ExprEval.ofComplex(
+ TYPE,
+ parsed
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ParseJsonExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return TYPE;
+ }
+ }
+ return new ParseJsonExpr(args);
+ }
+ }
+
+
+
+ public static class GetPathExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "get_path";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1PathPartsFromLiteral(name(), args);
+ class GetPathExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public GetPathExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new GetPathExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new GetPathExpr(args);
+ }
+ }
+
+ public static class JsonQueryExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_query";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonQueryExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonQueryExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.ofComplex(
+ TYPE,
+ NestedPathFinder.find(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonQueryExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // call all the output JSON typed
+ return TYPE;
+ }
+ }
+ return new JsonQueryExpr(args);
+ }
+ }
+
+ public static class JsonValueExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_value";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final List<NestedPathPart> parts = getArg1JsonPathPartsFromLiteral(name(), args);
+ class JsonValueExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonValueExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ return ExprEval.bestEffortOf(
+ NestedPathFinder.findLiteral(maybeUnwrapStructuredData(input), parts)
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new JsonValueExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ // we cannot infer the output type (well we could say it is 'STRING' right now because is all we support...
+ return null;
+ }
+ }
+ return new JsonValueExpr(args);
+ }
+ }
+
+ public static class ListPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "list_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class ListPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public ListPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ return ExprEval.ofType(
+ ExpressionType.STRING_ARRAY,
+ ImmutableList.copyOf(info.getLiteralFields())
+ );
+ }
+
+ @Override
+ public Expr visit(Shuttle shuttle)
+ {
+ List<Expr> newArgs = args.stream().map(x -> x.visit(shuttle)).collect(Collectors.toList());
+ return shuttle.visit(new ListPathsExpr(newArgs));
+ }
+
+ @Nullable
+ @Override
+ public ExpressionType getOutputType(InputBindingInspector inspector)
+ {
+ return ExpressionType.STRING_ARRAY;
+ }
+ }
+ return new ListPathsExpr(args);
+ }
+ }
+
+ public static class JsonPathsExprMacro implements ExprMacroTable.ExprMacro
+ {
+ public static final String NAME = "json_paths";
+
+ @Override
+ public String name()
+ {
+ return NAME;
+ }
+
+ @Override
+ public Expr apply(List<Expr> args)
+ {
+ final StructuredDataProcessor processor = new StructuredDataProcessor()
+ {
+ @Override
+ public int processLiteralField(String fieldName, Object fieldValue)
+ {
+ // do nothing, we only want the list of fields returned by this processor
+ return 0;
+ }
+ };
+
+ class JsonPathsExpr extends ExprMacroTable.BaseScalarMacroFunctionExpr
+ {
+ public JsonPathsExpr(List<Expr> args)
+ {
+ super(name(), args);
+ }
+
+ @Override
+ public ExprEval eval(ObjectBinding bindings)
+ {
+ ExprEval input = args.get(0).eval(bindings);
+ // maybe in the future ProcessResults should deal in PathFinder.PathPart instead of strings for fields
+ StructuredDataProcessor.ProcessResults info = processor.processFields(maybeUnwrapStructuredData(input));
+ List<String> transformed = info.getLiteralFields()
Review Comment:
have you thought about caching some results here? For example, many literal fields are going to be repeated across input records and `parseJqPath` doesn't seem cheap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12753:
URL: https://github.com/apache/druid/pull/12753#discussion_r932909483
##########
processing/src/main/java/org/apache/druid/segment/data/CompressedBlockReader.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.base.Preconditions;
+import org.apache.druid.collections.ResourceHolder;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.CompressedPools;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.IntBuffer;
+import java.util.function.Supplier;
+
+/**
+ * Reader for a virtual contiguous address range backed by compressed blocks of data.
+ *
+ * Format:
+ * | version (byte) | compression (byte) | num blocks (int) | block size (int) | end offsets | compressed data |
+ *
+ * This mechanism supports two modes of use, the first where callers may ask for a range of data from the underlying
+ * blocks, provided by {@link #getRange(long, int)}. The {@link ByteBuffer} provided by this method may or may not
+ * be valid after additional calls to {@link #getRange(long, int)} or calls to {@link #seekBlock(int)}.
+ *
+ * For fixed width values which are aligned with the block size, callers may also use the method
+ * {@link #getDecompressedDataBuffer()} to have direct access to the current uncompressed block, and use the methods
+ * {@link #loadBlock(long)} to load the correct block and translate a virtual offset into the relative offset, or
+ * {@link #seekBlock(int)} to change which block is currently loaded.
+ *
+ * {@link #getRange(long, int)} uses these same mechanisms internally to supply data.
+ */
+public final class CompressedBlockReader implements Closeable
+{
+ private static final ByteBuffer NULL_VALUE = ByteBuffer.wrap(new byte[0]);
+ public static final byte VERSION = 0x01;
+
+ public static Supplier<CompressedBlockReader> fromByteBuffer(ByteBuffer buffer, ByteOrder byteOrder)
+ {
+ byte versionFromBuffer = buffer.get();
+
+ if (versionFromBuffer == VERSION) {
+ final CompressionStrategy compression = CompressionStrategy.forId(buffer.get());
+ final int blockSize = buffer.getInt();
+ assert CompressedPools.BUFFER_SIZE == blockSize;
+ Preconditions.checkState(
+ blockSize <= CompressedPools.BUFFER_SIZE,
+ "Maximum block size must be less than " + CompressedPools.BUFFER_SIZE
+ );
+ final int numBlocks = buffer.getInt();
+ final int offsetsSize = numBlocks * Integer.BYTES;
+ // buffer is at start of ending offsets
+ final ByteBuffer offsets = buffer.asReadOnlyBuffer().order(byteOrder);
+ offsets.limit(offsets.position() + offsetsSize);
+ final IntBuffer offsetView = offsets.slice().order(byteOrder).asIntBuffer();
+ final int compressedSize = offsetView.get(numBlocks - 1);
+
+ // move to start of compressed data
+ buffer.position(buffer.position() + offsetsSize);
+ final ByteBuffer compressedData = buffer.asReadOnlyBuffer().order(byteOrder);
+ compressedData.limit(compressedData.position() + compressedSize);
+ buffer.position(buffer.position() + compressedSize);
+
+ final ByteBuffer compressedDataView = compressedData.slice().order(byteOrder);
+ return () -> new CompressedBlockReader(
+ compression,
+ numBlocks,
+ blockSize,
+ offsetView.asReadOnlyBuffer(),
+ compressedDataView.asReadOnlyBuffer().order(byteOrder),
+ byteOrder
+ );
+ }
+ throw new IAE("Unknown version[%s]", versionFromBuffer);
+ }
+
+ private final CompressionStrategy.Decompressor decompressor;
+
+ private final int numBlocks;
+ private final int div;
+ private final int rem;
+ private final IntBuffer endOffsetsBuffer;
+ private final ByteBuffer compressedDataBuffer;
+
+ private final ResourceHolder<ByteBuffer> decompressedDataBufferHolder;
+ private final ByteBuffer decompressedDataBuffer;
+
+ private final ByteOrder byteOrder;
+ private final Closer closer;
+ private int currentBlockNumber = -1;
+
+ public CompressedBlockReader(
+ CompressionStrategy compressionStrategy,
+ int numBlocks,
+ int blockSize,
+ IntBuffer endOffsetsBuffer,
+ ByteBuffer compressedDataBuffer,
+ ByteOrder byteOrder
+ )
+ {
+ this.decompressor = compressionStrategy.getDecompressor();
+ this.numBlocks = numBlocks;
+ this.div = Integer.numberOfTrailingZeros(blockSize);
+ this.rem = blockSize - 1;
+ this.endOffsetsBuffer = endOffsetsBuffer;
+ this.compressedDataBuffer = compressedDataBuffer;
+ this.closer = Closer.create();
+ this.decompressedDataBufferHolder = CompressedPools.getByteBuf(byteOrder);
+ closer.register(decompressedDataBufferHolder);
+ this.decompressedDataBuffer = decompressedDataBufferHolder.get();
+ this.decompressedDataBuffer.clear();
+ this.byteOrder = byteOrder;
+ }
+
+ /**
+ * Get size in bytes of virtual contiguous buffer
+ */
+ public long getSize()
Review Comment:
no, this is the end position of the last uncompressed value so the uncompressed size. The `endOffsetsBuffer` contains the uncompressed end position of every value that is compressed in the blocks. Using the uncompressed start and end position of a value and knowing the size of blocks we can then locate which block(s) the value is contained in to extract it.
I'll try to add some more comments and javadocs to this area so its a bit clearer what is going on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on PR #12753:
URL: https://github.com/apache/druid/pull/12753#issuecomment-1183571908
I created some query benchmarks to see how various nested column queries perform against traditional columns and see where things currently stand to help guide which areas I will be investigating in the future. I'll add the benchmark code in a follow-up PR. The nested columns are identical to the traditional columns, created using a transform expression:
```
json_object('long1', long1, 'nesteder', json_object('string1', string1, 'long2', long2, 'double3',double3))
```
For the most part things are pretty similar as expected, and the ones that aren't I will be digging deeper into. For numeric columns, which have indexes in nested columns, the indexes in most cases make the queries for nested columns faster, though in a few cases are currently slower. One of the double columns with the bound filter, which part of the reason for that is that currently there is no native numeric range index for the bound filter to use, so bound filters on numbers have to use a `DruidPredicateIndex` which has to match against every value in the dictionary instead of being able to take short-cuts like `LexicographicalRangeIndex` does for strings. I'll be adding this in a follow-up PR as well.
```
SELECT SUM(long1) FROM foo
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 0 5000000 false avgt 5 36.711 ± 0.917 ms/op
SqlNestedDataBenchmark.querySql 0 5000000 force avgt 5 15.587 ± 0.276 ms/op
SqlNestedDataBenchmark.querySql 1 5000000 false avgt 5 39.224 ± 0.870 ms/op
SqlNestedDataBenchmark.querySql 1 5000000 force avgt 5 15.877 ± 0.440 ms/op
SELECT SUM(long1), SUM(long2) FROM foo
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT)) FROM foo
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 2 5000000 false avgt 5 63.805 ± 1.036 ms/op
SqlNestedDataBenchmark.querySql 2 5000000 force avgt 5 30.381 ± 1.201 ms/op
SqlNestedDataBenchmark.querySql 3 5000000 false avgt 5 66.660 ± 0.806 ms/op
SqlNestedDataBenchmark.querySql 3 5000000 force avgt 5 30.341 ± 1.124 ms/op
SELECT SUM(long1), SUM(long2), SUM(double3) FROM foo
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 4 5000000 false avgt 5 78.570 ± 1.657 ms/op
SqlNestedDataBenchmark.querySql 4 5000000 force avgt 5 37.777 ± 1.295 ms/op
SqlNestedDataBenchmark.querySql 5 5000000 false avgt 5 82.672 ± 1.010 ms/op
SqlNestedDataBenchmark.querySql 5 5000000 force avgt 5 37.887 ± 0.802 ms/op
SELECT string1, SUM(long1) FROM foo GROUP BY 1 ORDER BY 2,
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo GROUP BY 1 ORDER BY 2,
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 6 5000000 false avgt 5 269.560 ± 1.454 ms/op
SqlNestedDataBenchmark.querySql 6 5000000 force avgt 5 157.090 ± 4.058 ms/op
SqlNestedDataBenchmark.querySql 7 5000000 false avgt 5 373.162 ± 2.871 ms/op
SqlNestedDataBenchmark.querySql 7 5000000 force avgt 5 195.213 ± 1.993 ms/op
SELECT string1, SUM(long1), SUM(double3) FROM foo GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo GROUP BY 1 ORDER BY 2
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 8 5000000 false avgt 5 251.743 ± 6.438 ms/op
SqlNestedDataBenchmark.querySql 8 5000000 force avgt 5 172.322 ± 14.814 ms/op
SqlNestedDataBenchmark.querySql 9 5000000 false avgt 5 417.454 ± 21.276 ms/op
SqlNestedDataBenchmark.querySql 9 5000000 force avgt 5 215.228 ± 9.304 ms/op
SELECT SUM(long1) FROM foo WHERE string1 = '10000' OR string1 = '1000'
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.string1') = '10000' OR JSON_VALUE(nested, '$.nesteder.string1') = '1000'
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 10 5000000 false avgt 5 11.482 ± 0.495 ms/op
SqlNestedDataBenchmark.querySql 10 5000000 force avgt 5 11.549 ± 0.303 ms/op
SqlNestedDataBenchmark.querySql 11 5000000 false avgt 5 11.695 ± 0.293 ms/op
SqlNestedDataBenchmark.querySql 11 5000000 force avgt 5 11.931 ± 0.338 ms/op
SELECT SUM(long1) FROM foo WHERE long2 = 10000 OR long2 = 1000
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) = 10000 OR JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) = 1000
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 12 5000000 false avgt 5 78.895 ± 2.158 ms/op
SqlNestedDataBenchmark.querySql 12 5000000 force avgt 5 48.814 ± 0.874 ms/op
SqlNestedDataBenchmark.querySql 13 5000000 false avgt 5 1.297 ± 0.008 ms/op
SqlNestedDataBenchmark.querySql 13 5000000 force avgt 5 1.277 ± 0.011 ms/op
SELECT SUM(long1) FROM foo WHERE double3 < 10000.0 AND double3 > 1000.0
SELECT SUM(JSON_VALUE(nested, '$.long1' RETURNING BIGINT)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) < 10000.0 AND JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) > 1000.0
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 14 5000000 false avgt 5 92.982 ± 1.473 ms/op
SqlNestedDataBenchmark.querySql 14 5000000 force avgt 5 54.729 ± 0.429 ms/op
SqlNestedDataBenchmark.querySql 15 5000000 false avgt 5 580.472 ± 28.064 ms/op
SqlNestedDataBenchmark.querySql 15 5000000 force avgt 5 561.494 ± 54.096 ms/op
SELECT long1, SUM(double3) FROM foo WHERE string1 = '10000' OR string1 = '1000' GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.long1' RETURNING BIGINT), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.string1') = '10000' OR JSON_VALUE(nested, '$.nesteder.string1') = '1000' GROUP BY 1 ORDER BY 2
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 16 5000000 false avgt 5 129.760 ± 9.953 ms/op
SqlNestedDataBenchmark.querySql 16 5000000 force avgt 5 133.015 ± 20.961 ms/op
SqlNestedDataBenchmark.querySql 17 5000000 false avgt 5 142.197 ± 8.773 ms/op
SqlNestedDataBenchmark.querySql 17 5000000 force avgt 5 132.048 ± 15.546 ms/op
SELECT string1, SUM(double3) FROM foo WHERE long2 < 10000 AND long2 > 1000 GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) < 10000 AND JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) > 1000 GROUP BY 1 ORDER BY 2
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 18 5000000 false avgt 5 161.445 ± 7.024 ms/op
SqlNestedDataBenchmark.querySql 18 5000000 force avgt 5 138.212 ± 19.673 ms/op
SqlNestedDataBenchmark.querySql 19 5000000 false avgt 5 123.486 ± 5.029 ms/op
SqlNestedDataBenchmark.querySql 19 5000000 force avgt 5 120.079 ± 6.822 ms/op
SELECT string1, SUM(double3) FROM foo WHERE double3 < 10000.0 AND double3 > 1000.0 GROUP BY 1 ORDER BY 2
SELECT JSON_VALUE(nested, '$.nesteder.string1'), SUM(JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE)) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) < 10000.0 AND JSON_VALUE(nested, '$.nesteder.double3' RETURNING DOUBLE) > 1000.0 GROUP BY 1 ORDER BY 2
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 20 5000000 false avgt 5 280.393 ± 15.369 ms/op
SqlNestedDataBenchmark.querySql 20 5000000 force avgt 5 174.545 ± 2.702 ms/op
SqlNestedDataBenchmark.querySql 21 5000000 false avgt 5 802.647 ± 32.078 ms/op
SqlNestedDataBenchmark.querySql 21 5000000 force avgt 5 591.274 ± 16.460 ms/op
SELECT long2 FROM foo WHERE long2 IN (1, 19, 21, 23, 25, 26, 46),
SELECT JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) IN (1, 19, 21, 23, 25, 26, 46),
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 22 5000000 false avgt 5 273.464 ± 15.731 ms/op
SqlNestedDataBenchmark.querySql 22 5000000 force avgt 5 272.270 ± 20.511 ms/op
SqlNestedDataBenchmark.querySql 23 5000000 false avgt 5 174.960 ± 1.923 ms/op
SqlNestedDataBenchmark.querySql 23 5000000 force avgt 5 177.920 ± 4.095 ms/op
SELECT long2 FROM foo WHERE long2 IN (1, 19, 21, 23, 25, 26, 46) GROUP BY 1",
SELECT JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) FROM foo WHERE JSON_VALUE(nested, '$.nesteder.long2' RETURNING BIGINT) IN (1, 19, 21, 23, 25, 26, 46) GROUP BY 1
Benchmark (query) (rowsPerSegment) (vectorize) Mode Cnt Score Error Units
SqlNestedDataBenchmark.querySql 24 5000000 false avgt 5 318.280 ± 7.544 ms/op
SqlNestedDataBenchmark.querySql 24 5000000 force avgt 5 210.866 ± 14.684 ms/op
SqlNestedDataBenchmark.querySql 25 5000000 false avgt 5 215.200 ± 2.366 ms/op
SqlNestedDataBenchmark.querySql 25 5000000 force avgt 5 152.399 ± 22.695 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
gianm commented on PR #12753:
URL: https://github.com/apache/druid/pull/12753#issuecomment-1183626156
Thanks for the details. It's helpful in understanding the maturity of the feature. It sounds relatively mature by the standards of a new feature.
> So, long story short, due to the unknowns I think the answer for right now is that operators should experiment with `COMPLEX<json>` columns to see if they work well for their use case, and use them in production if so, otherwise provide feedback so that we can continue to make improvements and expand the use cases this is good for?
To me this sounds like the basics of the feature are production-ready. There may be various callouts about performance, but it seems that the compatibility story is tight enough that the feature doesn't need the experimental markings at this time.
You mentioned being somewhat less certain about the behavior of nested arrays. We should figure out if that part is going to be included in the production-ready feature set, or if we'll call that particular scenario out as an evolving area. What is your intent & recommendation in this area?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on pull request #12753: druid nested data column type
Posted by GitBox <gi...@apache.org>.
clintropolis commented on PR #12753:
URL: https://github.com/apache/druid/pull/12753#issuecomment-1183687543
>To me this sounds like the basics of the feature are production-ready. There may be various callouts about performance, but it seems that the compatibility story is tight enough that the feature doesn't need the experimental markings at this time.
I think that is fair :+1:
>You mentioned being somewhat less certain about the behavior of nested arrays. We should figure out if that part is going to be included in the production-ready feature set, or if we'll call that particular scenario out as an evolving area. What is your intent & recommendation in this area?
It is definitely going to be an evolving area (which I think could be said of our array support in general), though there are probably a narrow range use cases that could be used today, mainly where array lengths and element positions are known and have some meaning and query time operations are primarily extracting and operating on individual elements. This is more or less the current limitations of `flattenSpec` with nested arrays I think.
There are some lower hanging fruit that would improve stuff in the near term, some of which might be possible to get in before the next release. The first supporting wildcards in the subset of the path syntax that we support, which would allow `JSON_QUERY` and `JSON_VALUE` (or something like it.. i'm not sure entirely how the `RETURNING` syntax would work with array types in SQL so need to do some tinkering there) to extract complete arrays. For `JSON_QUERY` these results would still be `COMPLEX<json>` typed , but `JSON_VALUE`* would spit out druid literal array types (`ARRAY<LONG>`, `ARRAY<STRING>`, etc).
For nested arrays of JSON objects extracted by `JSON_QUERY`, i think we will want a way to convert a `COMLEX<json>` into an `ARRAY<COMPLEX<json>>` so that they too can take part in array operations, _especially_ once we add a native `UNNEST` function to transform arrays into tables, which would be the path to exploding out these nested objects and performing operations on their contents.
At some point after that, I intend to introduce the option to begin storing literal arrays in nested `ARRAY` typed columns instead of them broken out into separate columns for individual elements like they currently exist (so that array operations don't have to decompress a bunch of separate columns to do stuff).
I guess I'm getting a bit into the weeds, but my point I guess is that I think this feature will evolve along-side and should help us improve array support in general, so am hyped to get it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org