You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2024/01/05 12:03:11 UTC
[PR] overhaul DruidPredicateFactory to better handle 3VL (druid)
clintropolis opened a new pull request, #15629:
URL: https://github.com/apache/druid/pull/15629
### Description
fixes some bugs caused by some limitations of the original design of how DruidPredicateFactory interacts with 3-value logic. The primary impacted area was with how filters on values transformed with expressions or extractionFn which turn non-null values into nulls, which were not possible to be modelled with the 'isNullInputUnknown' method
changes:
* adds DruidObjectPredicate to specialize string, array, and object based predicates instead of using guava Predicate
* DruidPredicateFactory now uses DruidObjectPredicate
* introduces DruidPredicateMatch enum, which all predicates returned from DruidPredicateFactory now use instead of booleans to indicate match. This means DruidLongPredicate, DruidFloatPredicate, DruidDoublePredicate, and the newly added DruidObjectPredicate apply methods all now return DruidPredicateMatch. This allows matchers and indexes
* isNullInputUnknown has been removed from DruidPredicateFactory
#### Release note
_tbd_
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] overhaul DruidPredicateFactory to better handle 3VL (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm merged PR #15629:
URL: https://github.com/apache/druid/pull/15629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] overhaul DruidPredicateFactory to better handle 3VL (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #15629:
URL: https://github.com/apache/druid/pull/15629#discussion_r1443292306
##########
processing/src/main/java/org/apache/druid/query/filter/InDimFilter.java:
##########
@@ -302,10 +304,14 @@ public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
if (indexSupplier == null) {
// column doesn't exist, match against null
- return Filters.makeMissingColumnNullIndex(
- predicateFactory.makeStringPredicate().apply(null),
- selector
- );
+ DruidPredicateMatch match = predicateFactory.makeStringPredicate().apply(null);
Review Comment:
heh i did that at first but were only the 3 callers so seemed barely worth having, then again after sleeping on it i agree, it is just enough callers so will add it back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] overhaul DruidPredicateFactory to better handle 3VL (druid)
Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15629:
URL: https://github.com/apache/druid/pull/15629#discussion_r1442818018
##########
benchmarks/src/test/java/org/apache/druid/benchmark/DimensionPredicateFilterBenchmark.java:
##########
@@ -73,18 +74,13 @@
new DruidPredicateFactory()
{
@Override
- public Predicate<String> makeStringPredicate()
+ public DruidObjectPredicate<String> makeStringPredicate()
{
- return new Predicate<String>()
- {
- @Override
- public boolean apply(String input)
- {
- if (input == null) {
- return false;
- }
- return Integer.parseInt(input) % 2 == 0;
+ return (DruidObjectPredicate<String>) input -> {
+ if (input == null) {
+ return DruidPredicateMatch.UNKNOWN;
}
+ return DruidPredicateMatch.of(Integer.parseInt(input) % 2 == 0);
Review Comment:
## Missing catch of NumberFormatException
Potential uncaught 'java.lang.NumberFormatException'.
[Show more details](https://github.com/apache/druid/security/code-scanning/6418)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] overhaul DruidPredicateFactory to better handle 3VL (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #15629:
URL: https://github.com/apache/druid/pull/15629#discussion_r1443074801
##########
processing/src/main/java/org/apache/druid/query/filter/DruidObjectPredicate.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import javax.annotation.Nullable;
+
+public interface DruidObjectPredicate<T>
+{
+ static <T> DruidObjectPredicate<T> alwaysFalse()
Review Comment:
This isn't really the right name, since it's not always false. So, we should change it to make the call sites more legible. Maybe a better name is `falseWithUnknownForNull()`?
##########
processing/src/main/java/org/apache/druid/query/filter/vector/ArrayVectorValueMatcher.java:
##########
@@ -77,16 +77,12 @@ public ReadableVectorMatch match(final ReadableVectorMatch mask, boolean include
for (int i = 0; i < mask.getSelectionSize(); i++) {
final int rowNum = mask.getSelection()[i];
Object o = vector[rowNum];
- if (includeUnknown && o == null && predicateFactory.isNullInputUnknown()) {
- selection[numRows++] = rowNum;
- } else if (o == null || o instanceof Object[]) {
- if (predicate.apply((Object[]) o)) {
- selection[numRows++] = rowNum;
- }
- } else {
- if (predicate.apply(new Object[]{o})) {
+ if ((o == null || o instanceof Object[])) {
Review Comment:
The new code is definitely more readable!
##########
processing/src/main/java/org/apache/druid/query/filter/DruidPredicateMatch.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+/**
+ * Three-value logic result for matching values with predicates produced by {@link DruidPredicateFactory}
+ *
+ * @see DruidObjectPredicate
+ * @see DruidLongPredicate
+ * @see DruidFloatPredicate
+ * @see DruidDoublePredicate
+ */
+public enum DruidPredicateMatch
+{
+ /**
+ * Value does not match
+ */
+ FALSE,
+ /**
+ * Value matches
+ */
+ TRUE,
+ /**
+ * Value is unknown to match, for example from a null input to a predicate which does not match null as true or false
+ */
+ UNKNOWN;
+
+ /**
+ * Convenience method for {@link ValueMatcher} and {@link org.apache.druid.query.filter.vector.VectorValueMatcher}
+ * implementations to pass through the 'includeUnknown' parameter to the predicate match result.
+ */
+ public boolean matches(boolean includeUknown)
Review Comment:
`includeUnknown` (spelling)
##########
processing/src/main/java/org/apache/druid/query/filter/InDimFilter.java:
##########
@@ -302,10 +304,14 @@ public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
if (indexSupplier == null) {
// column doesn't exist, match against null
- return Filters.makeMissingColumnNullIndex(
- predicateFactory.makeStringPredicate().apply(null),
- selector
- );
+ DruidPredicateMatch match = predicateFactory.makeStringPredicate().apply(null);
Review Comment:
Move this logic into `Filters.makeMissingColumnNullIndex`, and change its signature to `makeMissingColumnNullIndex(DruidPredicateMatch matchesNull, final ColumnIndexSelector selector)`?
Then this code would be a bit simpler, it would remain a call to `Filters.makeMissingColumnNullIndex`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] overhaul DruidPredicateFactory to better handle 3VL (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #15629:
URL: https://github.com/apache/druid/pull/15629#discussion_r1443291517
##########
processing/src/main/java/org/apache/druid/query/filter/DruidObjectPredicate.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import javax.annotation.Nullable;
+
+public interface DruidObjectPredicate<T>
+{
+ static <T> DruidObjectPredicate<T> alwaysFalse()
Review Comment:
I guess I was trying to match the naming/behavior of `DruidLongPredicate.ALWAYS_FALSE` and other numeric predicates which also return unknown for `applyNull`, should I rename them all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] overhaul DruidPredicateFactory to better handle 3VL (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #15629:
URL: https://github.com/apache/druid/pull/15629#discussion_r1443303683
##########
processing/src/main/java/org/apache/druid/query/filter/DruidObjectPredicate.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import javax.annotation.Nullable;
+
+public interface DruidObjectPredicate<T>
+{
+ static <T> DruidObjectPredicate<T> alwaysFalse()
Review Comment:
I would, yes. It's not good for a thing named "always false" to not always return false, since people will get confused and use it where it isn't appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org