You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/01/11 03:30:41 UTC
[GitHub] [iceberg] kbendick opened a new pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
kbendick opened a new pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062
Adds support for a `NOT_STARTS_WITH` operator and closes https://github.com/apache/iceberg/issues/1952.
This also ensures that pushdown happens when evaluating Parquet dictionaries as well as Parquet row groups. It also ensures that Spark will push this filter down, which is particularly important for queries to remove string partition columns, especially in the case of the identity partition spec or in the truncation partition spec when truncation length is less than or equal to the notStartsWith predicate term.
I've added quite a number of tests. Admittedly, many of them were added in order to aide my own understanding of the codebase so that I could better contribute in the future. So please feel free to suggest any that should be removed in order to spare CI running time and cut down on potential code rot.
I also added a few tests around `startsWith`, which I'd be happy to factor out into their own PR. I'm adding some comments to explain my reasoning for the changes.
cc @shardulm94 @RussellSpitzer @rdblue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-770344943
Thanks everybody for the reviews so far. I know it's not a tiny PR so I really appreciate it.
I will update everything that's been mentioned. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554894135
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
notStartsWith can't necessarily be stood in for !startsWith depending on the situation.
Here's a quote from @rdblue on the original ticket.
```
Yes, I think we will need a NotStartsWith to handle this.
We can't keep the Not after rewriting because some evaluation doesn't support negation. For example, when lower and upper bounds are used to evaluate whether to read a file, the result may be true to indicate that the value may be contained in the file. That should not be negated because it does not confirm that the value is definitely in the file: if looking for files that "do not contain X", we can't use "not(file might contain X)".
```
However, in situations where we believe it can be, I would agree the reducing unneeded code is a good way to go. But given that we're dealing with ROWS_MIGHT_MATCH, my understanding is that we're not necessarily able to negate that as all rows don't necessarily match. So rows in a file could very well be `ROWS_MIGHT_MATCH` for startsWith as well as for notStartsWith.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r583264473
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -284,8 +284,30 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
if (predicate.isUnaryPredicate()) {
return Expressions.predicate(predicate.op(), name);
- } else if (predicate.isLiteralPredicate()) {
- return ProjectionUtil.truncateArray(name, predicate.asLiteralPredicate(), this);
+ } else if (predicate instanceof BoundLiteralPredicate) {
Review comment:
Why change this to `instanceof` instead of using `isLiteralPredicate`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567175054
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Can you add a test case for `all_nulls`? I think that should always match a `notStartsWith` filter because there are no values. You should be able to detect that because the lower and upper bounds are `null`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567172999
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
Review comment:
The inclusive projection needs similar handling to my comment about strict projection. If the width of the truncate transform is greater than the width of the `startsWith` or `notStartsWith` prefix, then the original predicate can be returned directly. If the width is the same as the prefix, then `startsWith` should be converted to `equals` and `notStartsWith` should be converted to `notEquals`. If the width is greater, then `ProjectionUtil.truncateArray` can be used to truncate the prefix for `startsWith`, but not the negated version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554924932
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -69,6 +69,8 @@ public boolean test(T value) {
return cmp.compare(value, literal.value()) != 0;
case STARTS_WITH:
return String.valueOf(value).startsWith((String) literal.value());
+ case NOT_STARTS_WITH:
+ return !String.valueOf(value).startsWith((String) literal.value());
Review comment:
It's also tempting to just use the comparator from literal, like follows:
```java
cmp.compare(value, literal.value()) == 0;
```
However, because we're only interested in checking whether or not `value` does not start with the predicate's literal term, we would need to truncate like we do elsewhere so I don't think that the simple one line above would be correct (elegant as it would be).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555233694
##########
File path: api/src/main/java/org/apache/iceberg/expressions/StrictMetricsEvaluator.java
##########
@@ -402,6 +402,11 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_NOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ROWS_MIGHT_NOT_MATCH;
Review comment:
This is correct, but I think there are cases where this can definitively conclude that rows cannot match. For example, `notStartsWith("x")` when the bounds are `["a", "b"]`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571705897
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
##########
@@ -500,6 +534,48 @@ public void testUnpartitionedStartsWith() {
Assert.assertEquals("junction", matchedData.get(0));
}
+ @Test
+ public void testUnpartitionedNotStartsWith() {
+ Dataset<Row> df = spark.read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(unpartitioned.toString());
+
+ List<String> matchedData = df.select("data")
+ .where("data NOT LIKE 'jun%'")
+ .as(Encoders.STRING())
+ .collectAsList();
+
+ List<String> expected = testRecords(SCHEMA).stream()
+ .map(r -> r.getField("data").toString())
+ .filter(d -> !d.startsWith("jun"))
+ .collect(Collectors.toList());
+
+ Assert.assertEquals(9, matchedData.size());
+ Assert.assertEquals(new HashSet<>(expected), new HashSet<>(matchedData));
Review comment:
Let me know if you'd like me to remove any further tests from here.
I removed the one that duplicates this (but uses the scala spark sql functions). I kept the tests that mirror the various partitioned by data and partitioned by id tests for `startsWith`. If you think the latter two tests are not needed (`testPartitionedByIdNotStartsWith` and `testPartitionedByDataNotStartsWithFilter`), please let me know and I'd be happy to remove them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r609322064
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -439,6 +446,44 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("all_same_value_or_null", "a"), SPEC, false).eval(FILE);
+ Assert.assertFalse("Should skip: range doesn't match", shouldRead);
Review comment:
I have updated this to return `true` and have also gone over all of the `notStartsWith` implementations to ensure that they handle `null` input elements properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567177398
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -373,6 +373,25 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_CANNOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Boolean hasNonDictPage = isFallback.get(id);
+ if (hasNonDictPage == null || hasNonDictPage) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ Set<T> dictionary = dict(id, lit.comparator());
+ for (T item : dictionary) {
+ if (!item.toString().startsWith(lit.value().toString())) {
+ return ROWS_MIGHT_MATCH;
+ }
Review comment:
It would probably be good to follow up with a change to use comparators, but this should be okay for now. It isn't in a tight loop (row groups are usually >= 128MB) and it short-circuits quickly in most cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r762686988
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -242,6 +248,10 @@ public R or(R leftResult, R rightResult) {
throw new UnsupportedOperationException("Unsupported operation.");
}
+ public <T> R notStartsWith(Bound<T> expr, Literal<T> lit) {
+ throw new UnsupportedOperationException("Unsupported operation.");
Review comment:
I think generally we try to touch as few lines as needed in some cases (or avoid unnecessary changes), as that makes it easier on people who maintain forks.
I don't disagree with you that the error message should be more descriptive, but given what I know about the API and the implemented classes, I don't think these specific base class for `BoundVisitor` actually get called anywhere. Notice how many of them return `null` as well.
As for the specific error message, it's just to be consistent with the existing one for `startsWith`.
Given the size of the PR already, I think it's best to keep unrelated changes out and in a separate PR (particularly updating `startsWith` - but until then, let's be consistent).
Large PRs like this already face a hurdle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-998332867
Note: We'll want to add a test in here if the other PR gets approved: https://github.com/apache/iceberg/pull/3757/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772705145
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -285,7 +285,29 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
if (predicate.isUnaryPredicate()) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate.isLiteralPredicate()) {
- return ProjectionUtil.truncateArray(name, predicate.asLiteralPredicate(), this);
+ BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
Review comment:
Another way of thinking about this is that the truncate transform doesn't affect the result of `startsWith` if truncation is longer than the `startsWith` prefix. If I'm looking for strings that start with a 2-character pattern, like `aa`, then truncating strings to 4 characters first doesn't matter.
That's why both `STARTS_WITH` and `NOT_STARTS_WITH` have the same logic here and in the strict projection. As long as the truncation length is >= the prefix length, the original `startsWith` predicate is still valid. (Though we can optimize a little if we know that the string will be no longer than the prefix.)
So the only cases we need to worry about are when the truncated value is shorter than the prefix.
* For `startsWith` and `inclusive`, using a shortened prefix will find values that might match
* For `notStartsWith` and `inclusive`, using a shortened value will exclude values that do match, so there is no projection
* For `startsWith` and `strict`, using a shortened value would include values that do not match, so there is no projection
* For `notStartsWith` and `strict`, using a shortened value will exclude all values that must match
The tests have some good examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571705583
##########
File path: spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetReadMetadataColumns.java
##########
@@ -77,6 +77,9 @@
private static final int NUM_ROWS = 1000;
private static final List<InternalRow> DATA_ROWS;
private static final List<InternalRow> EXPECTED_ROWS;
+ // We filter for $"data" notStartsWith str1, along the row group splits, to ensure the row group
Review comment:
Looking intp it more, the reason I originally put this test in this file is because there are some tests that specifically operate along the splits already.
What I'm looking to test here is that the filter is properly applied to row groups - by checking that the spark parquet reader returns all rows from a given split if any row in that split matches the filter.
However, I agree that that's not the purpose of this test file. I'm going to move this test to its own file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554910739
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
One way to reduce the code bloat / repetition would be to define a function `private boolean startsWith(ByteBuffer value, ByteBuffer prefix, Comparator<ByteBuffer> cmp)` (or possibly returning int to allow it to be used inside of the `startsWith` code), but that would ultimately just remove the repetition of the three lines of code that truncate the upper / lower bound to match the byte length of the prefix and then compare. The functions would still need to be different from each other.
I considered doing that in this PR, but I felt that changing the implementation of `startsWith` at the same time would possibly be a bit too many changes for one PR (as this PR is already large). I'd be happy to clean it up to make the code a bit cleaner, especially in a follow up PR, but I don't think that we can use `!startsWith(ref, lit)` for really any of these cases other than `ResidualEvaluator` unfortunately 🙁 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567173516
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,24 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc => true", evaluator.eval(TestHelpers.Row.of("abc")));
Review comment:
Usually, the expectation in context is given with "should". That way, when it fails the message is "abc startsWith abc should be true". It's easy to read and understand the problem that way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567177020
##########
File path: core/src/test/java/org/apache/iceberg/TestManifestReaderStats.java
##########
@@ -82,6 +88,13 @@ public void testReadWithFilterIncludesFullStats() throws IOException {
ManifestEntry<DataFile> entry = entries.iterator().next();
assertFullStats(entry.file());
}
+
+ try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest, FILE_IO)
+ .filterRows(Expressions.notStartsWith("data", "Z"))) {
+ CloseableIterable<ManifestEntry<DataFile>> entries = reader.entries();
+ ManifestEntry<DataFile> entry = entries.iterator().next();
+ assertFullStats(entry.file());
+ }
Review comment:
I think this is testing something else, so this file doesn't need to change. Unless I'm missing something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-782361915
Looking into what's been done since I opened this, Flink now has filter pushdown. So we likely need to update `org.apache.iceberg.flink.FlinkFilters` to account for NOT_STARTS_WITH. Or at the least we might consider adding some tests in Flink that use `NOT LIKE` to ensure those queries will work.
I will look into that asap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r556179867
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -373,6 +373,25 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_CANNOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Boolean hasNonDictPage = isFallback.get(id);
+ if (hasNonDictPage == null || hasNonDictPage) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ Set<T> dictionary = dict(id, lit.comparator());
+ for (T item : dictionary) {
+ if (!item.toString().startsWith(lit.value().toString())) {
+ return ROWS_MIGHT_MATCH;
+ }
Review comment:
As discussed elsewhere in this PR, we'd like to keep the code similar to the existing semantics. I'm going to resolve this comment to make it easier for others to digest this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r559381026
##########
File path: site/docs/api.md
##########
@@ -172,6 +172,7 @@ Supported predicate expressions are:
* `in`
* `notIn`
* `startsWith`
+* `notStartsWith`
Review comment:
I added this to the expressions listed in the API docs, though I'm not sure how often we publish the docs. Likely this should not be reflected in the documentation until a version of iceberg is released with `NOT_STARTS_WITH` in it.
If we publish the docs when we publish a new version, then this should be fine to add now. Otherwise, I'll remove it and add a new PR that can be merged in at the appropriate time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r770852322
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ // notStartsWith will match unless all values must start with the prefix. This happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
+ // if lower is shorter than the prefix, it can't start with the prefix
Review comment:
I think let's try to keep pronouns out of this since I have difficulty parsing this statement. I'm always confused though.
// if lower is shorter than the prefix then lower doesn't start with the prefix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772724033
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -373,6 +373,25 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_CANNOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Boolean hasNonDictPage = isFallback.get(id);
+ if (hasNonDictPage == null || hasNonDictPage) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ Set<T> dictionary = dict(id, lit.comparator());
+ for (T item : dictionary) {
+ if (!item.toString().startsWith(lit.value().toString())) {
+ return ROWS_MIGHT_MATCH;
+ }
Review comment:
Unresolving this as a reminder to myself to follow up on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772721963
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -285,7 +285,29 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
if (predicate.isUnaryPredicate()) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate.isLiteralPredicate()) {
- return ProjectionUtil.truncateArray(name, predicate.asLiteralPredicate(), this);
+ BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
Review comment:
Yeah that is a correct summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772657012
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
Sure. Originally, when this was worked on, you had said to leave it up to the engine to decide because we don't implement SQL 3-boolean logic.
I'm ok with your argument presented here though. Have any input on how to update the comment then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator (including Spark 3.2)
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383141
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,24 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc => true", evaluator.eval(TestHelpers.Row.of("abc")));
Review comment:
Yeah. i used a similar format to what already exists in some of the other tests, but I didn't really like it here or either of them too much. It made it harder to see what part is needing truncation etc and it doesn't follow the very standard "should" type statements.
I agree we should update these to use should. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567178793
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
##########
@@ -469,6 +473,21 @@ public void testPartitionedByDataStartsWithFilter() {
Assert.assertEquals(1, reader.planInputPartitions().size());
}
+ @Test
+ public void testPartitionedByDataNotStartsWithFilter() {
+ File location = buildPartitionedTable("partitioned_by_data", PARTITION_BY_DATA, "data_ident", "data");
+
+ DataSourceOptions options = new DataSourceOptions(ImmutableMap.of(
+ "path", location.toString())
+ );
Review comment:
It seems weird to have one `)` on the same line and a second on its own line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555358109
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -69,6 +69,8 @@ public boolean test(T value) {
return cmp.compare(value, literal.value()) != 0;
case STARTS_WITH:
return String.valueOf(value).startsWith((String) literal.value());
+ case NOT_STARTS_WITH:
+ return !String.valueOf(value).startsWith((String) literal.value());
Review comment:
Cool. I agree about being worried re: changing past behavior. Wasn't entirely sure of all of the conditions around it (I'm not a UTF-8 expert admittedly), but wanted to bring it up. I'll close this comment then! Thanks for clarification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554903095
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
`ResidualEvaluator` is a case where we could simply return `!startsWith(ref, lit)`. That's actually how I defined it (using the same ternary expression by swapping the return values on true or false). But given that function body for `ResidualEvaluator` is only one line, it's arguable whether or not that's worth it as it could bring confusion about which cases can we use `!startsWith` as a stand in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571701029
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
Review comment:
I've updated the inclusive projection as discussed earlier in the file: https://github.com/apache/iceberg/pull/2062/files#diff-402f9a8b8744486a0283fed354bca6a2aa6f3eae7c116a07d08d0c67fccf4077R287-R310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383832
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Will do.
I had thought about the matching on `notStartsWith` for null boundaries, but I think I originally did not check properly. I just checked and nulls are matched when using Spark.
```
scala> val df = Seq("hi", null, "there", "world").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
df.selectExpr("word not like 'wo%'").show(false)
+-------------------+
|(NOT word LIKE wo%)|
+-------------------+
|true |
|null |
|true |
|false |
+-------------------+
scala> df.select(not($"word".startsWith("wo"))).show
+--------------------------+
|(NOT startswith(word, wo))|
+--------------------------+
| true|
| null|
| true|
| false|
+--------------------------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r559263109
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##########
@@ -151,5 +151,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
public <T> Boolean startsWith(Bound<T> valueExpr, Literal<T> lit) {
return ((String) valueExpr.eval(struct)).startsWith((String) lit.value());
}
+
+ @Override
+ public <T> Boolean notStartsWith(Bound<T> valueExpr, Literal<T> lit) {
+ return !((String) valueExpr.eval(struct)).startsWith((String) lit.value());
Review comment:
Update
For `ResidualEvaluator`, it can't be `!startsWith(ref, lit)` as `ResidualEvaluator` returns an expression, so it would have to be `startsWith(ref, lit).negate()`. This works since we're returning `alwaysTrue()` or `alwaysFalse()`, but it is a little confusing (and potentially error prone in my opinion in the longer term). Also, in the same file, we don't define `notIn` by using `in` in this way. I personally think `ResidualEvaluator` should be left as is.
Additionally, `BoundLiteralPredicate` cannot be defined in terms of `STARTS_WITH` as the function signature is just `test(T value)` and then the operation is caught in a switch statement and the operation is stored as a value on `this` (the `BoundLiteralPredicate` instance). So we'd have to return `negate().test(value)` which is also pretty confusing and likely error prone.
I'm going to close this conversation as it seems that the instances where we can easily and simply replace `notStartsWith` with `!starsWith` have all been covered. Feel free to reopen it or comment further if you disagree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-782425002
> I will look into that asap.
Let's do it in a follow-up, this PR is already very large.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571680008
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -108,6 +112,9 @@ public void testAllNulls() {
shouldRead = ManifestEvaluator.forRowFilter(startsWith("all_nulls", "asad"), SPEC, true).eval(FILE);
Assert.assertFalse("Should skip: startsWith on all null column", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("all_nulls", "asad"), SPEC, true).eval(FILE);
+ Assert.assertFalse("Should skip: notStartsWith on all null column", shouldRead);
Review comment:
In line with the definition of `notStartsWith` against `null` had with @rdblue in the `testStringNotStartsWith` method being added to this file, we've decided to define `notStartsWith` aginast `null` to be `true`. In this case, `true` means that the file is maybe a match. Because Iceberg does not implement SQL's 3-way boolean logic, we have to make a choice here. It makes rational sense to say that `null` does not start with some non-null string, and returning `true` here allows for the query engine to then further process the file and make its own decisions according to its own customs (such as whether the query engine uses SQL 3-way boolean logic or not). If we return less data, the query engine does not have the opportunity to return all possible results.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r563401055
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
I'm going to mark this as closed to make it easier for people to review this. As discused in a comment elsewhere with @rdblue and @RussellSpitzer, places where substitution of `notStartsWith`'s definition as `!startsWith` have been updated. The remaining ones require the usage of a `notStartsWith` specific definition due to the above contraints in regards to what is being tested for (i.e. we're testing in those cases for `file might match`).
If anybody has any concerns feel free to comment here or elsewhere 👍
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
I'm going to mark this conversation as resolved to make it easier for people to review this. As discused in a comment elsewhere with @rdblue and @RussellSpitzer, places where substitution of `notStartsWith`'s definition as `!startsWith` have been updated. The remaining ones require the usage of a `notStartsWith` specific definition due to the above contraints in regards to what is being tested for (i.e. we're testing in those cases for `file might match`).
If anybody has any concerns feel free to comment here or elsewhere 👍
##########
File path: site/docs/api.md
##########
@@ -172,6 +172,7 @@ Supported predicate expressions are:
* `in`
* `notIn`
* `startsWith`
+* `notStartsWith`
Review comment:
Marking this as resolved to make this easier to review as this is not a concern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r557860156
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -91,6 +93,8 @@ public String toString() {
return term() + " != " + literal;
case STARTS_WITH:
return term() + " startsWith \"" + literal + "\"";
+ case NOT_STARTS_WITH:
+ return term() + " notStartsWith \"" + literal + "\"";
Review comment:
I'll resolve this conversation so as to make the PR more digestible to others, feel free to open it back up if you think this needs to be dealt with _in this PR_.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r565083422
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
+ if (isStringPrefixPredicate(pred)) {
if (pred.literal().value().length() < width()) {
Review comment:
Seems missing `if (pred.literal().value().length() > width())` case for `NOT_STARTS_WITH`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554897025
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
Here's the ticket for your reference: https://github.com/apache/iceberg/issues/1952
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567382673
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
Review comment:
Ok. I can definitely do that. A lot of the code style that I used was more or less chosen to mirror `startsWith` (so that both could be refactored together if need be).
I much prefer the commented version as well. I chose to leave it on the slimmer side with respect to comments as there are not necessarily a ton surrounding it, but I do agree that this makes it much easier to reason about (as well as eensure we avoid unnecessary allocations, etc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554894135
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
notStartsWith can't necessarily be stood in for not(startsWith) depending on the situation.
Here's a quote from @rdblue on the original ticket.
```
Yes, I think we will need a NotStartsWith to handle this.
We can't keep the Not after rewriting because some evaluation doesn't support negation. For example, when lower and upper bounds are used to evaluate whether to read a file, the result may be true to indicate that the value may be contained in the file. That should not be negated because it does not confirm that the value is definitely in the file: if looking for files that "do not contain X", we can't use "not(file might contain X)".
```
However, in situations where we believe it can be, I would agree the reducing unneeded code is a good way to go. But given that we're dealing with ROWS_MIGHT_MATCH, my understanding is that we're not necessarily able to negate that as all rows don't necessarily match. So rows in a file could very well be `ROWS_MIGHT_MATCH` for startsWith as well as for notStartsWith.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r563401055
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
I'm going to mark this conversation as resolved to make it easier for people to review this. As discused in a comment elsewhere with @rdblue and @RussellSpitzer, places where substitution of `notStartsWith`'s definition as `!startsWith` have been updated. The remaining ones require the usage of a `notStartsWith` specific definition due to the above contraints in regards to what is being tested for (i.e. we're testing in those cases for `file might match`).
If anybody has any concerns feel free to comment here or elsewhere 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772656080
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ // notStartsWith will match unless all values must start with the prefix. This happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
+ // if lower is shorter than the prefix, it can't start with the prefix
+ if (lower.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ int cmp = comparator.compare(BinaryUtil.truncateBinary(lower, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ ByteBuffer upper = upperBounds.get(id);
+ // if upper is shorter than the prefix, it can't start with the prefix
+ if (upper.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ cmp = comparator.compare(BinaryUtil.truncateBinary(upper, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ // both bounds match the prefix, so all rows must match the prefix and therefore do not satisfy
+ // the predicate
+ return ROWS_CANNOT_MATCH;
+ }
+ }
+ }
+
+ return ROWS_MIGHT_MATCH;
+ }
+
+ private boolean mayContainNull(Integer id) {
Review comment:
This may not be needed if you instead change to use `containsNullsOnly`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772652476
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
I don't think this is correct. In SQL, `null LIKE 'a%'` should result in `null`, and `null NOT LIKE 'a%'` should also result in `null`. So in both cases, `null` does not match and is ignored.
Here, I think this should use `containsNullsOnly(id)`. In that case, we know that no rows match and can exit before checking bounds:
```java
if (containsNullsOnly(id)) {
return ROWS_CANNOT_MATCH;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772652476
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
I don't think this is correct. In SQL, `null LIKE 'a%'` should result in `null`, and `null NOT LIKE 'a%'` should also result in `null`. So in both cases, `null` does not match and is ignored.
Here, I think this should use `containsNullsOnly(id)`. In that case, we know that no rows match and can exit before checking bounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator (including Spark 3.2)
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-998376616
Thanks, @kbendick! Great to have this in before 0.13.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-996200065
I'd like to do another pass at this, since this is a pretty delicate area of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-996089510
> I'm interested in getting this in soon, @rdblue did you have any remaining issues with the PR?
I think @rdblue mentioned he was going to try to find time to review this again (so it’s on his radar). If you can review and approve it or provide any remaining feedback, that would help move things along a good amount @RussellSpitzer 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-996082780
I'm interested in getting this in soon, @rdblue did you have any remaining issues with the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-985925145
@cccs-eric @rdblue this PR is ready for review again.
I only implemented tests against Spark 3.2. Let me know if you'd like me to add the tests for the other versions of Spark in this PR as well.
I've rebased off of master. Once we decide this is good, I'll rebase again and squash this into fewer commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-782361915
Looking into what's been done since I opened this, Flink now has filter pushdown. So we likely need to update `org.apache.iceberg.flink.FlinkFilters` to account for NOT_STARTS_WITH.
I will look into that asap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554928604
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
The other place where `!startsWith` would be correct is in the current implementation of `BoundLiteralPredicate`, though again that's a simple one line (and I've also marked down some concerns I have with the implementation as it currently is written). In the case of `BoundLiteralPredicate` though, regardless of the final implementation of `startsWith`, since we're not dealing with `file might contain x` but just a single value comparison, `!startsWith` would be correct. But again, it's only one line and we don't save much in terms of complexity.
Outside of that, it seems that substituting `not(startsWith)` wouldn't work. As mentioned, most of the time we've asssumed that any `NOT` nodes have been removed from the AST of the query via `RewriteNot`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r556179318
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
Review comment:
Given it's not checked elsewhere, I'm resolving this comment to make the PR more digestible to others.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571683130
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveMetricsEvaluator.java
##########
@@ -529,6 +533,47 @@ public void testStringStartsWith() {
Assert.assertFalse("Should not read: range doesn't match", shouldRead);
}
+ @Test
+ // More interesting test cases are included in TestNotStartsWith
+ public void testStringNotStartsWith() {
+ boolean shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "a"), true).eval(FILE);
+ Assert.assertTrue("Should read: no stats", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "a"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aa"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aaa"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "1s"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "1str1x"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "ff"), true).eval(FILE_4);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aB"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "dWX"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "5"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "3str3x"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ String aboveMax = UnicodeUtil.truncateStringMax(Literal.of("イロハニホヘト"), 4).value().toString();
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", aboveMax), true).eval(FILE_4);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+ }
Review comment:
Resolving this conversation to make this PR easier to review. Feel free to reopen it if there is concern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r609321044
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -439,6 +446,44 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("all_same_value_or_null", "a"), SPEC, false).eval(FILE);
+ Assert.assertFalse("Should skip: range doesn't match", shouldRead);
Review comment:
I have updated this and also gone over all of the `notStartsWith` implementations to ensure that they handle `null` appropriately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567175366
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestPredicateBinding.java
##########
@@ -108,6 +110,25 @@ public void testComparisonPredicateBinding() {
}
}
+ @Test
+ @SuppressWarnings("unchecked")
+ public void tesPredicateBindingForStringPrefixComparisons() {
Review comment:
Typo: `tesPredicate...` should be `testPredicate...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567173278
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,24 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc => true", evaluator.eval(TestHelpers.Row.of("abc")));
+ Assert.assertFalse("abc startsWith Abc => false", evaluator.eval(TestHelpers.Row.of("Abc")));
+ Assert.assertFalse("a startsWith abc => false", evaluator.eval(TestHelpers.Row.of("a")));
Review comment:
Can you also add a tests for `abcd startsWith abc`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567386194
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
+ if (isStringPrefixPredicate(pred)) {
if (pred.literal().value().length() < width()) {
Review comment:
Cool. That's much cleaner. And I see now how/why the starts with case can't make as many strict guarantees due to truncation. Thank you both. I will update this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r770906869
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -285,7 +285,29 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
if (predicate.isUnaryPredicate()) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate.isLiteralPredicate()) {
- return ProjectionUtil.truncateArray(name, predicate.asLiteralPredicate(), this);
+ BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
Review comment:
So if I understand this we have a set of optimizations here,
```
If we have a starts with:
If our prefix is shorter than the truncate:
Then behave has normal and compare with normal StartsWith
If our prefix is the same length as the truncate:
Then we can do an equal instead
If our prefix is longer than the truncate:
Truncate the literal and check that
If we have a notStartsWith:
If our prefix is shorter than the truncate:
Behave as normal with notStartsWith
If our prefix is the same length as truncate:
Compare with Equals
If our prefix is longer than the truncate:
No pruning possible because we don't have enough information
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r770927862
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -368,6 +368,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
}
Set<T> dictionary = dict(id, lit.comparator());
+ if (dictionary.contains(null)) {
Review comment:
In the other locations a comment is also included
// Allow query engine to make its own decisions regarding SQL 3-valued boolean logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772689105
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ // notStartsWith will match unless all values must start with the prefix. This happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
+ // if lower is shorter than the prefix, it can't start with the prefix
+ if (lower.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ int cmp = comparator.compare(BinaryUtil.truncateBinary(lower, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ ByteBuffer upper = upperBounds.get(id);
+ // if upper is shorter than the prefix, it can't start with the prefix
+ if (upper.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ cmp = comparator.compare(BinaryUtil.truncateBinary(upper, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ // both bounds match the prefix, so all rows must match the prefix and therefore do not satisfy
+ // the predicate
+ return ROWS_CANNOT_MATCH;
+ }
+ }
+ }
+
+ return ROWS_MIGHT_MATCH;
+ }
+
+ private boolean mayContainNull(Integer id) {
Review comment:
Nevermind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-986424127
Thanks for the reviews @cccs-eric! I added the test cases and otherwise responded. I agree we should update the exception messages and will either do it in this PR or a follow up PR.
Let me know if you have any other feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r563401055
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
I'm going to mark this as closed to make it easier for people to review this. As discused in a comment elsewhere with @rdblue and @RussellSpitzer, places where substitution of `notStartsWith`'s definition as `!startsWith` have been updated. The remaining ones require the usage of a `notStartsWith` specific definition due to the above contraints in regards to what is being tested for (i.e. we're testing in those cases for `file might match`).
If anybody has any concerns feel free to comment here or elsewhere 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555233694
##########
File path: api/src/main/java/org/apache/iceberg/expressions/StrictMetricsEvaluator.java
##########
@@ -402,6 +402,11 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_NOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ROWS_MIGHT_NOT_MATCH;
Review comment:
This is correct, but I think there are cases where this can definitively conclude that rows cannot match. For example, `notStartsWith("x")` when the bounds are `["a", "b"]`. We can add that in a follow-up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554903095
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
`ResidualEvaluator` is a case where we could simply return `!startsWith(ref, lit)`. That's actually basically how I defined it (using the same ternary expression but swapping the return values on true or false). That's because the `ResidualEvaluator` definitions can only return `alwaysTrue()` or `alwaysFalse()`. But given that the function body for `ResidualEvaluator` is only one line, it's arguable whether or not that's worth it as it could bring confusion about which cases can we use `!startsWith` as a stand in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554894135
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
notStartsWith can't necessarily be replaced with !startsWith depending on the situation and the semantics of each evaluator.
Here's a quote from @rdblue on the original ticket.
```
Yes, I think we will need a NotStartsWith to handle this.
We can't keep the Not after rewriting because some evaluation doesn't support negation.
For example, when lower and upper bounds are used to evaluate whether to read a file,
the result may be true to indicate that the value may be contained in the file.
That should not be negated because it does not confirm that the value is definitely in the file:
if looking for files that "do not contain X", we can't use "not(file might contain X)".
```
However, in situations where it can be, I would agree that reducing unneeded code is a good way to go. But given that we're dealing with `ROWS_MIGHT_MATCH`, my understanding is that we're not necessarily able to negate that as all rows don't necessarily match. So rows in a file could very well be `ROWS_MIGHT_MATCH` for startsWith as well as for notStartsWith.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568253490
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
##########
@@ -469,6 +473,21 @@ public void testPartitionedByDataStartsWithFilter() {
Assert.assertEquals(1, reader.planInputPartitions().size());
}
+ @Test
+ public void testPartitionedByDataNotStartsWithFilter() {
+ File location = buildPartitionedTable("partitioned_by_data", PARTITION_BY_DATA, "data_ident", "data");
+
+ DataSourceOptions options = new DataSourceOptions(ImmutableMap.of(
+ "path", location.toString())
+ );
Review comment:
Agreed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567178697
##########
File path: spark2/src/main/java/org/apache/iceberg/spark/SparkFilters.java
##########
@@ -135,6 +136,13 @@ public static Expression convert(Filter filter) {
case NOT:
Not notFilter = (Not) filter;
+ Filter sparkChild = notFilter.child();
+ // Special case to use the Iceberg NOT_STARTS_WITH operator for
+ // Not(StringStartsWith) to allow filter push down.
+ if (sparkChild instanceof StringStartsWith) {
+ StringStartsWith castedChild = (StringStartsWith) sparkChild;
+ return notStartsWith(castedChild.attribute(), castedChild.value());
+ }
Review comment:
I don't think this is necessary. It's fine to convert `not(startsWith(...))` to the same structure. The important thing is that when we rewrite to remove `not`, we can produce a `notStartsWith` instead of failing.
We probably don't need to touch Spark at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555359253
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ResidualEvaluator.java
##########
@@ -214,6 +214,11 @@ public Expression or(Expression leftResult, Expression rightResult) {
return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysTrue() : alwaysFalse();
}
+ @Override
+ public <T> Expression notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysFalse() : alwaysTrue();
+ }
Review comment:
Ok. I agree on fixing it later (if necessary). And the `CharSequence` recommendation seems much more sane, as that's all the API allows for and then we reduce allocations.
Thanks for the info, Ryan.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554894135
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
notStartsWith can't necessarily be stood in for !startsWith depending on the situation.
Here's a quote from @rdblue on the original ticket.
```
Yes, I think we will need a NotStartsWith to handle this.
We can't keep the Not after rewriting because some evaluation doesn't support negation.
For example, when lower and upper bounds are used to evaluate whether to read a file,
the result may be true to indicate that the value may be contained in the file.
That should not be negated because it does not confirm that the value is definitely in the file:
if looking for files that "do not contain X", we can't use "not(file might contain X)".
```
However, in situations where it can be, I would agree that reducing unneeded code is a good way to go. But given that we're dealing with `ROWS_MIGHT_MATCH`, my understanding is that we're not necessarily able to negate that as all rows don't necessarily match. So rows in a file could very well be `ROWS_MIGHT_MATCH` for startsWith as well as for notStartsWith.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555137388
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Expression.java
##########
@@ -74,6 +75,10 @@ public Operation negate() {
return Operation.NOT_IN;
case NOT_IN:
return Operation.IN;
+ case STARTS_WITH:
+ return Operation.NOT_STARTS_WITH;
Review comment:
oh nevermind this is negate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383832
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Will do.
I had thought about the matching on `notStartsWith` for null boundaries, but I think I originally did not check properly (I was using a filter function I think). I just went through and checked with SQL, and the nulls matched on the equivalent for `notStartsWith`.
```
scala> val df = Seq("hi", null, "there", "world").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
df.selectExpr("word not like 'wo%'").show(false)
+-------------------+
|(NOT word LIKE wo%)|
+-------------------+
|true |
|null |
|true |
|false |
+-------------------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555139230
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -116,7 +116,11 @@ public R or(R leftResult, R rightResult) {
}
public <T> R startsWith(BoundReference<T> ref, Literal<T> lit) {
- throw new UnsupportedOperationException("Unsupported operation.");
+ throw new UnsupportedOperationException("startsWith operation is not supported by the visitor");
Review comment:
nit: startsWitch expression
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555231693
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ResidualEvaluator.java
##########
@@ -214,6 +214,11 @@ public Expression or(Expression leftResult, Expression rightResult) {
return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysTrue() : alwaysFalse();
}
+ @Override
+ public <T> Expression notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysFalse() : alwaysTrue();
+ }
Review comment:
If we can't use String comparison, then the right way to compare is to convert to `CharSequence`. `ByteBuffer` gets converted to `CharBuffer` using UTF-8 and that implements `CharSequence`, while `String` already implements `CharSequence`.
We know that the literal's value is a `String` because that is the only literal value that is accepted by the API, so that cast is fine here and in other places. I'm not sure that the result of evaluating the reference is a String. It seems like that should be a `CharSequence` instead. But let's fix that problem later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r589252718
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -439,6 +446,44 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("all_same_value_or_null", "a"), SPEC, false).eval(FILE);
+ Assert.assertFalse("Should skip: range doesn't match", shouldRead);
Review comment:
I believe that either there's a code path that wasn't updated when we made the decision to return files / row groups etc that evaluated as possibly passing the predicate.
I will review those places.
The other thing that comes to mind is that possibly the file is not fully mocked for the given situaton, but I don't think that's the case. I'm pretty sure a path was not updated after we made the decision that a predicate x notStartsWith a string literal for x is true (to deal with SQL 3-way boolean logic which Iceberg does not implement).
I'll review this as soon as I have time and update.
Good catch! Thank you :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554920714
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -69,6 +69,8 @@ public boolean test(T value) {
return cmp.compare(value, literal.value()) != 0;
case STARTS_WITH:
return String.valueOf(value).startsWith((String) literal.value());
+ case NOT_STARTS_WITH:
+ return !String.valueOf(value).startsWith((String) literal.value());
Review comment:
The other way to handle this to be similar to the rest of the cases and to use the `Comparator` that `literal` brings with it would be to convert the `T` to a ByteBuffer, and then truncate it and check equality with the ByteBuffer representation of `literal`, as follows.
```java
ByteBuffer valueAsBytes = Conversions.toByteBuffer(ref().type(), value);
ByteBuffer literalAsBytes = literal.toByteBuffer();
int length = Math.min(valueAsBytes.remaining(), literalAsBytes.remaining());
Comparators.unsignedBytes(BinaryUtil.truncateBinary(valueAsBytes, length), prefixAsBytes)) == 0;
```
Depending on the Type `T`, in the above code snippet we might need to use `Comparators.naturalOrder` instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554894135
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
notStartsWith can't necessarily be stood in for !startsWith depending on the situation.
Here's a quote from @rdblue on the original ticket.
```
Yes, I think we will need a NotStartsWith to handle this.
We can't keep the Not after rewriting because some evaluation doesn't support negation.
For example, when lower and upper bounds are used to evaluate whether to read a file,
the result may be true to indicate that the value may be contained in the file.
That should not be negated because it does not confirm that the value is definitely in the file:
if looking for files that "do not contain X", we can't use "not(file might contain X)".
```
However, in situations where we believe it can be, I would agree the reducing unneeded code is a good way to go. But given that we're dealing with ROWS_MIGHT_MATCH, my understanding is that we're not necessarily able to negate that as all rows don't necessarily match. So rows in a file could very well be `ROWS_MIGHT_MATCH` for startsWith as well as for notStartsWith.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567176708
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestTruncatesResiduals.java
##########
@@ -179,5 +180,12 @@ public void testStringTruncateTransformResiduals() {
assertResidualValue(spec, startsWith("value", "bcd"), "ab", Expression.Operation.FALSE);
assertResidualPredicate(spec, startsWith("value", "bcd"), "bc");
assertResidualValue(spec, startsWith("value", "bcd"), "cd", Expression.Operation.FALSE);
+
+ // not starts with
+ assertResidualValue(spec, notStartsWith("value", "bc"), "ab", Expression.Operation.TRUE);
+ assertResidualValue(spec, notStartsWith("value", "bc"), "bc", Expression.Operation.FALSE);
+ assertResidualPredicate(spec, notStartsWith("value", "bcd"), "ab");
+ assertResidualValue(spec, notStartsWith("value", "bcd"), "bc", Expression.Operation.FALSE);
+ assertResidualPredicate(spec, notStartsWith("value", "bcd"), "cd");
Review comment:
I think these depend on the projections, so they may need to be updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r563401212
##########
File path: site/docs/api.md
##########
@@ -172,6 +172,7 @@ Supported predicate expressions are:
* `in`
* `notIn`
* `startsWith`
+* `notStartsWith`
Review comment:
Marking this as resolved to make this easier to review as this is not a concern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772695468
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -285,7 +285,29 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
if (predicate.isUnaryPredicate()) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate.isLiteralPredicate()) {
- return ProjectionUtil.truncateArray(name, predicate.asLiteralPredicate(), this);
+ BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
Review comment:
Yes, I think that's an accurate summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772731732
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
Removed the comment entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772720707
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java
##########
@@ -473,6 +473,66 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+ Long valueCount = valueCounts.get(id);
+
+ // Iceberg does not implement SQL 3-boolean logic. Therefore, for all null values, we have decided to
+ // return ROWS_MIGHT_MATCH in order to allow the query engine to further evaluate this partition, as
+ // null does not start with any non-null value.
+ if (valueCount == null) {
+ // the column is not present and is all nulls
+ return ROWS_MIGHT_MATCH;
+ }
+
+ Statistics<Binary> colStats = (Statistics<Binary>) stats.get(id);
+ if (colStats != null && !colStats.isEmpty()) {
+ if (mayContainNull(colStats)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ if (hasNonNullButNoMinMax(colStats, valueCount)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefix = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ Binary lower = colStats.genericGetMin();
+ // notStartsWith will match unless all values must start with the prefix. this happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lower != null) {
Review comment:
Here as well, we may want to validate that both `lower` and `upper` are non-null before doing any comparison, but this is very minor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772725693
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
How does this look?
```java
// Iceberg expressions implement normal logic and ignore 3-value logic. Null does not start with any prefix.
// If there is a null value then, that value doesn't start with the given prefix and would match the expression.
if (mayContainNull(id)) {
return ROWS_MIGHT_MATCH;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cccs-eric commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
cccs-eric commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-983578166
I am very interested by what this PR is fixing since I am running into this problem (see [comment](https://github.com/apache/iceberg/issues/1952#issuecomment-983573754)). Is it possible to finish the work started and merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r762686988
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -242,6 +248,10 @@ public R or(R leftResult, R rightResult) {
throw new UnsupportedOperationException("Unsupported operation.");
}
+ public <T> R notStartsWith(Bound<T> expr, Literal<T> lit) {
+ throw new UnsupportedOperationException("Unsupported operation.");
Review comment:
I think generally we try to touch as few lines as needed in some cases (or avoid unnecessary changes), as that makes it easier on people who maintain forks.
I don't disagree with you that the error message should be more descriptive, but given what I know about the API, I don't think these specific base class for `BoundVisitor` actually gets called. Notice how many of them return `null` as well.
Here, it's just to be consistent with the existing one. If other's feel like it should be changed, I'm happy to do so, but I think given the size of the PR it's best to keep unrelated changes (particularly updating `startsWith`) in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r762687037
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,27 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abc")));
Review comment:
Good call. Will add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-985925145
@cccs-eric @rdblue this PR is ready for review again and rebased off of master.
I only implemented tests against Spark 3.2. Let me know if you'd like me to add the tests for the other versions of Spark in this PR as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-985880725
> @cccs-eric, this is something we should complete. I didn't realize we hadn't finished it. @kbendick can you update this when you get a chance?
Yeah sure. I'd be happy to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-985925145
@cccs-eric @rdblue this PR is ready for review again.
I only implemented tests against Spark 3.2. Let me know if you'd like me to add the tests for the other versions of Spark in this PR as well.
I've rebased offf of master. Once we decide this is goood, I'll rebase again and squash this into fewer commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-985880725
> @cccs-eric, this is something we should complete. I didn't realize we hadn't finished it. @kbendick can you update this when you get a chance?
Yeah sure. I'd be happy to. I'll take care of that now and update to use Spark 3.2 and the new path structures. Most everything else should be more or less the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555362955
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -116,7 +116,11 @@ public R or(R leftResult, R rightResult) {
}
public <T> R startsWith(BoundReference<T> ref, Literal<T> lit) {
- throw new UnsupportedOperationException("Unsupported operation.");
+ throw new UnsupportedOperationException("startsWith operation is not supported by the visitor");
Review comment:
So I noticed that a few of the other surrounding functions use `operation` as well. Namely, `IN` and `NOT_IN`, in addition to `STARTS_WITH` and now `NOT_STARTS_WITH`. I think it's likely because of throwing `UnsupportedOperationException`. I can update the others as well while I'm in here 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r583270380
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -439,6 +446,44 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("all_same_value_or_null", "a"), SPEC, false).eval(FILE);
+ Assert.assertFalse("Should skip: range doesn't match", shouldRead);
Review comment:
Why doesn't this return `true`? If there are nulls, then at least null does not start with `a`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r556190456
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ResidualEvaluator.java
##########
@@ -214,6 +214,11 @@ public Expression or(Expression leftResult, Expression rightResult) {
return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysTrue() : alwaysFalse();
}
+ @Override
+ public <T> Expression notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysFalse() : alwaysTrue();
+ }
Review comment:
I'm going to resolve this conversation, as we've decided to fix this later, if need be. This is to make the PR more digestible for others. Feel free to open it back up if you'd like.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567180246
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/SparkFilters.java
##########
@@ -157,6 +158,13 @@ public static Expression convert(Filter filter) {
case NOT:
Not notFilter = (Not) filter;
+ Filter sparkChild = notFilter.child();
+ // Special case to use the Iceberg NOT_STARTS_WITH operator for
+ // Not(StringStartsWith) to allow filter push down.
+ if (sparkChild instanceof StringStartsWith) {
+ StringStartsWith stringStartsWith = (StringStartsWith) sparkChild;
+ return notStartsWith(stringStartsWith.attribute(), stringStartsWith.value());
+ }
Review comment:
Same comment as the other one above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555223207
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##########
@@ -151,5 +151,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
public <T> Boolean startsWith(Bound<T> valueExpr, Literal<T> lit) {
return ((String) valueExpr.eval(struct)).startsWith((String) lit.value());
}
+
+ @Override
+ public <T> Boolean notStartsWith(Bound<T> valueExpr, Literal<T> lit) {
+ return !((String) valueExpr.eval(struct)).startsWith((String) lit.value());
Review comment:
I agree here. `!startsWith()` is good because it avoids implementing the same thing twice.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] shangxinli commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
shangxinli commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554696316
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
Review comment:
if notStartsWith can be negated from startsWith, why not replace the whole method body with "return !startsWith(ref, lit);", instead of replicating so much code?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick removed a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick removed a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-759143183
I think a timeout or something caused the Java 8 tests to fail. Running them locally, they all pass (and the Java 11 tests pass as well).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772725693
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
How does this look?
```java
// Iceberg expressions implement normal logic and ignore 3-value logic. Null does not start with any prefix.
// If there is a null value then, that value doesn't start with the given prefix and would match the expression.
if (mayContainNull(id)) {
return ROWS_MIGHT_MATCH;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r770906869
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -285,7 +285,29 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
if (predicate.isUnaryPredicate()) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate.isLiteralPredicate()) {
- return ProjectionUtil.truncateArray(name, predicate.asLiteralPredicate(), this);
+ BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
Review comment:
So if I understand this we have a set of optimizations here,
If we have a starts with:
If our prefix is shorter than the truncate:
Then behave has normal and compare with normal StartsWith
If our prefix is the same length as the truncate:
Then we can do an equal instead
If our prefix is longer than the truncate:
Truncate the literal and check that
If we have a notStartsWith:
If our prefix is shorter than the truncate:
Behave as normal with notStartsWith
If our prefix is the same length as truncate:
Compare with Equals
If our prefix is longer than the truncate:
No pruning possible because we don't have enough information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-985925145
@cccs-eric @rdblue this PR is ready for review again.
I only implemented tests against Spark 3.2. Let me know if you'd like me to add the tests for the other versions of Spark in this PR as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r565083422
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
+ if (isStringPrefixPredicate(pred)) {
if (pred.literal().value().length() < width()) {
Review comment:
Hmm, seems missing `if pred.literal().value().length() > width()` case for `NOT_STARTS_WITH`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571507967
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
Review comment:
I'm going to resolve this conversation to make this PR easier to consume. Please feel free to reopen it if you have concerns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568261124
##########
File path: core/src/test/java/org/apache/iceberg/TestManifestReaderStats.java
##########
@@ -82,6 +88,13 @@ public void testReadWithFilterIncludesFullStats() throws IOException {
ManifestEntry<DataFile> entry = entries.iterator().next();
assertFullStats(entry.file());
}
+
+ try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest, FILE_IO)
+ .filterRows(Expressions.notStartsWith("data", "Z"))) {
+ CloseableIterable<ManifestEntry<DataFile>> entries = reader.entries();
+ ManifestEntry<DataFile> entry = entries.iterator().next();
+ assertFullStats(entry.file());
+ }
Review comment:
You're right. I think this is one of the cases where I added in several tests in order to better understand how everything was executed (occasionally hopping through with the debugger). I'll remove the changes to this file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567175294
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveMetricsEvaluator.java
##########
@@ -529,6 +533,47 @@ public void testStringStartsWith() {
Assert.assertFalse("Should not read: range doesn't match", shouldRead);
}
+ @Test
+ // More interesting test cases are included in TestNotStartsWith
+ public void testStringNotStartsWith() {
+ boolean shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "a"), true).eval(FILE);
+ Assert.assertTrue("Should read: no stats", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "a"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aa"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aaa"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "1s"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "1str1x"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "ff"), true).eval(FILE_4);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aB"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "dWX"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "5"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "3str3x"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ String aboveMax = UnicodeUtil.truncateStringMax(Literal.of("イロハニホヘト"), 4).value().toString();
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", aboveMax), true).eval(FILE_4);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+ }
Review comment:
This seems fine to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568175971
##########
File path: api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java
##########
@@ -197,6 +197,8 @@ private ProjectionUtil() {
return predicate(Expression.Operation.EQ, name, transform.apply(boundary));
case STARTS_WITH:
return predicate(Expression.Operation.STARTS_WITH, name, transform.apply(boundary));
+ case NOT_STARTS_WITH:
+ return predicate(Expression.Operation.NOT_STARTS_WITH, name, transform.apply(boundary));
Review comment:
Ok. Should I just remove this case statement entirely then and allow the default block to catch it? I'm working on some tests as well so I'll push when I am done with that / in between busy times at work. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567175996
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
+ private static final Schema SCHEMA = new Schema(FIELD);
+
+ // All 50 rows have someStringCol = 'bbb', none are null (despite being optional).
+ private static final DataFile FILE_1 = new TestDataFile("file_1.avro", Row.of(), 50,
+ // any value counts, including nulls
+ ImmutableMap.of(1, 50L),
+ // null value counts
+ ImmutableMap.of(1, 0L),
+ // nan value counts
+ null,
+ // lower bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")),
+ // upper bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")));
+
+ @Test
+ public void testTruncateProjections() {
Review comment:
I think the projection tests need to be updated after fixing projections.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555136762
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##########
@@ -151,5 +151,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
public <T> Boolean startsWith(Bound<T> valueExpr, Literal<T> lit) {
return ((String) valueExpr.eval(struct)).startsWith((String) lit.value());
}
+
+ @Override
+ public <T> Boolean notStartsWith(Bound<T> valueExpr, Literal<T> lit) {
+ return !((String) valueExpr.eval(struct)).startsWith((String) lit.value());
Review comment:
we could just run !startsWith()
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554674081
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -69,6 +69,8 @@ public boolean test(T value) {
return cmp.compare(value, literal.value()) != 0;
case STARTS_WITH:
return String.valueOf(value).startsWith((String) literal.value());
+ case NOT_STARTS_WITH:
+ return !String.valueOf(value).startsWith((String) literal.value());
Review comment:
Should we be using the predefined comparators here? This seems like it could get iffy with non-ascii values given the difference between UTF-8 and Java's default String encoding of UTF-16.
I chose to go with the existing style here, but I'm not 100% convinced this is the best solution as opposed to using a comparator.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555366265
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##########
@@ -151,5 +151,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
public <T> Boolean startsWith(Bound<T> valueExpr, Literal<T> lit) {
return ((String) valueExpr.eval(struct)).startsWith((String) lit.value());
}
+
+ @Override
+ public <T> Boolean notStartsWith(Bound<T> valueExpr, Literal<T> lit) {
+ return !((String) valueExpr.eval(struct)).startsWith((String) lit.value());
Review comment:
Ok. That sounds good to me. As I mentioned above to @shangxinli there are a few places where !startsWith can be substituted.
As I see it currently, they are `ResidualEvaluator` and `BoundLiteralPredicate`. Should I do the same thing there?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554920714
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -69,6 +69,8 @@ public boolean test(T value) {
return cmp.compare(value, literal.value()) != 0;
case STARTS_WITH:
return String.valueOf(value).startsWith((String) literal.value());
+ case NOT_STARTS_WITH:
+ return !String.valueOf(value).startsWith((String) literal.value());
Review comment:
The other way to handle this to be similar to the rest of the cases and to use the `Comparator` that `literal` brings with it would be to convert the `T` to a ByteBuffer, and then truncate it and check equality with the ByteBuffer representation of `literal`, as follows.
```java
ByteBuffer valueAsBytes = Conversions.toByteBuffer(ref().type(), value);
ByteBuffer literalAsBytes = literal.toByteBuffer();
int length = Math.min(valueAsBytes.remaining(), literalAsBytes.remaining());
Comparators.unsignedBytes(BinaryUtil.truncateBinary(valueAsBytes, length), prefixAsBytes)) == 0;
```
Or if we wanted to just use the comparator from literal, we might consider the following. However, for `CharSequence` etc we would need to truncate like we do elsewhere so I don't think that this would be correct.
```java
cmp.compare(value, literal.value()) == 0;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567168725
##########
File path: api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java
##########
@@ -197,6 +197,8 @@ private ProjectionUtil() {
return predicate(Expression.Operation.EQ, name, transform.apply(boundary));
case STARTS_WITH:
return predicate(Expression.Operation.STARTS_WITH, name, transform.apply(boundary));
+ case NOT_STARTS_WITH:
+ return predicate(Expression.Operation.NOT_STARTS_WITH, name, transform.apply(boundary));
Review comment:
I don't think that this is correct.
This is called to project (inclusive) a `notStartsWith` predicate, like `notStartsWith("abc")`. If the truncate transform is truncating to 2 characters, then this would produce `notStartsWith("ab")`. The definition of an inclusive projection is that the projected predicate must match if the original predicate would match. That's violated here: "abd" matches the original predicate because it does not start with "abc". But, it does _not_ match the projected predicate. "abd" does start with "ab", so it would be rejected.
I think that the inclusive projection should return `null` for `NOT_STARTS_WITH`.
On the other hand, the strict projection could use this. The strict projection matches only if the original predicate _must_ match. Using the example from above, you can see that any string that does not start with "ab" also does not start with "abc".
I think you need to move this case to the `truncateArrayStrict` method.
We also need a few tests for projection in `TestTruncatesProjection`, starting with the example above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r569877701
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java
##########
@@ -472,6 +472,50 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Long valueCount = valueCounts.get(id);
+ if (valueCount == null) {
+ // the column is not present and is all nulls
+ return ROWS_CANNOT_MATCH;
+ }
+
+ Statistics<Binary> colStats = (Statistics<Binary>) stats.get(id);
+ if (colStats != null && !colStats.isEmpty()) {
+ if (hasNonNullButNoMinMax(colStats, valueCount)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ if (!colStats.hasNonNullValue()) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ Binary lower = colStats.genericGetMin();
+ // truncate lower bound so that its length in bytes is not greater than the length of prefix
+ int lowerLength = Math.min(prefixAsBytes.remaining(), lower.length());
+ int lowerCmp = comparator.compare(BinaryUtil.truncateBinary(lower.toByteBuffer(), lowerLength), prefixAsBytes);
+ if (lowerCmp == 0) {
+ Binary upper = colStats.genericGetMax();
+ // truncate upper bound so that its length in bytes is not greater than the length of prefix
+ int upperLength = Math.min(prefixAsBytes.remaining(), upper.length());
+ ByteBuffer upperByteBuffer = upper.toByteBuffer();
+ int upperCmp = comparator.compare(BinaryUtil.truncateBinary(upperByteBuffer, upperLength), prefixAsBytes);
+ if (upperCmp == 0) {
+ return ROWS_CANNOT_MATCH;
+ }
+ }
+ }
Review comment:
I have updated it as discussed elsewhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r557859899
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -91,6 +93,8 @@ public String toString() {
return term() + " != " + literal;
case STARTS_WITH:
return term() + " startsWith \"" + literal + "\"";
+ case NOT_STARTS_WITH:
+ return term() + " notStartsWith \"" + literal + "\"";
Review comment:
I think it's best that we follow up on this once it's merged. I'd prefer to not change too much of the existing behavior so as to give this PR the most opportunity to get approved and merged in.
Then we can evaluate whether or not the toString representation is over quoted. Thankfully, it does not appear that the toString representation of the bound literal term affects the results of the operation, at least based on my running through several of the tests with the debugger.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567382902
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
Review comment:
Cool. That makes sense. It did seem to me that there were some issues wrt width, but I was hesitant to alter too much of the existing code without having it reviewed by others. I will apply the changes you've suggested. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r569859335
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
I had some concerns about my original Spark based check, so I checked again. I was mistaken when I initially checked to see what Spark returns, as Spark implements SQL 3-way boolean logic, for example:
```
scala> val df = Seq("aa", "bb", null, "b").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
scala> df.createOrReplaceTempView("t")
scala> df.select(not($"word".startsWith("bb"))).show
+--------------------------+
|(NOT startswith(word, bb))|
+--------------------------+
| true|
| false|
| null|
| true|
+--------------------------+
scala> spark.sql("SELECT * FROM t WHERE word NOT LIKE 'bb%'").show
+----+
|word|
+----+
| aa|
| b|
+----+
```
However, Iceberg does not implement 3-way boolean logic. So we need to make a decision on whether or not we want Iceberg to return `true` or `false` in this case.
I discussed this offline with @rdblue and we've come up with the following rationale for an `all_nulls` file matching a `notStartsWith` filter:
The argument in favor of returning `true` is stronger than deciding to return `false`, i.e. by saying that null does not start with "bb" or whatever the predicate's value is. The reason for this is that:
(1) Logically, null does not start with "bb" (or any non-null value)
(2) If we have to choose between Iceberg surfacing too many values or too few, we would want Iceberg to return more values and then allow the query engine to make the final decision on the null value. If we return too few values, the query engine might not return the results that it would have if the underlying table format were not Iceberg. However, if we return too many values from the predicate filters, as in this case `true` means maybe and so the query engine can still ensure that the result is correct (according to the query engines definition of correct). When `false` means definitely not, the query engine has no ability to catch anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567175957
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
+ private static final Schema SCHEMA = new Schema(FIELD);
+
+ // All 50 rows have someStringCol = 'bbb', none are null (despite being optional).
+ private static final DataFile FILE_1 = new TestDataFile("file_1.avro", Row.of(), 50,
+ // any value counts, including nulls
+ ImmutableMap.of(1, 50L),
+ // null value counts
+ ImmutableMap.of(1, 0L),
+ // nan value counts
+ null,
+ // lower bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")),
+ // upper bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")));
+
+ @Test
+ public void testTruncateProjections() {
+ PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).truncate(COLUMN, 4).build();
+
+ assertProjectionInclusive(spec, notStartsWith(COLUMN, "ab"), "ab", Expression.Operation.NOT_STARTS_WITH);
+ assertProjectionInclusive(spec, notStartsWith(COLUMN, "abab"), "abab", Expression.Operation.NOT_STARTS_WITH);
+ assertProjectionInclusive(spec, notStartsWith(COLUMN, "ababab"), "abab", Expression.Operation.NOT_STARTS_WITH);
+
+ assertProjectionStrict(spec, notStartsWith(COLUMN, "ab"), "ab", Expression.Operation.NOT_STARTS_WITH);
+ assertProjectionStrict(spec, notStartsWith(COLUMN, "abab"), "abab", Expression.Operation.NOT_EQ);
+
+ Expression projection = Projections.strict(spec).project(notStartsWith(COLUMN, "abcde"));
+ Assert.assertTrue(projection instanceof False);
+ }
+
+ @Test
+ public void testTruncateStringWhenProjectedPredicateTermIsLongerThanWidth() {
+ Truncate<String> trunc = Truncate.get(Types.StringType.get(), 2);
+ Expression expr = notStartsWith(COLUMN, "abcde");
+ BoundPredicate<String> boundExpr = (BoundPredicate<String>) Binder.bind(SCHEMA.asStruct(), expr, false);
+
+ UnboundPredicate<String> projected = trunc.project(COLUMN, boundExpr);
+ Evaluator evaluator = new Evaluator(SCHEMA.asStruct(), projected);
+
+ Assert.assertFalse("notStartsWith(abcde, truncate(abcde,2)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcde")));
+
+ Assert.assertFalse("notStartsWith(abcde, truncate(ab, 2)) => false",
+ evaluator.eval(TestHelpers.Row.of("ab")));
+
+ Assert.assertFalse("notStartsWith(abcde, truncate(abcdz, 2)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcdz")));
+
+ Assert.assertTrue("notStartsWith(abcde, truncate(a, 2)) => true",
+ evaluator.eval(TestHelpers.Row.of("a")));
+
+ Assert.assertTrue("notStartsWith(abcde, truncate(abzcde, 2)) => true",
+ evaluator.eval(TestHelpers.Row.of("azcde")));
+ }
+
+ @Test
+ public void testTruncateStringWhenProjectedPredicateTermIsShorterThanWidth() {
+ Truncate<String> trunc = Truncate.get(Types.StringType.get(), 16);
+ Expression expr = notStartsWith(COLUMN, "ab");
+ BoundPredicate<String> boundExpr = (BoundPredicate<String>) Binder.bind(SCHEMA.asStruct(), expr, false);
+ UnboundPredicate<String> projected = trunc.project(COLUMN, boundExpr);
+ Evaluator evaluator = new Evaluator(SCHEMA.asStruct(), projected);
+
+ Assert.assertFalse("notStartsWith(ab, truncate(abcde, 16)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcde")));
+
+ Assert.assertFalse("notStartsWith(ab, truncate(ab, 16)) => false",
+ evaluator.eval(TestHelpers.Row.of("ab")));
+
+ Assert.assertTrue("notStartsWith(ab, truncate(a, 16)) => true",
+ evaluator.eval(TestHelpers.Row.of("a")));
+ }
+
+ @Test
+ public void testTruncateStringWhenProjectedPredicateTermIsEqualToWidth() {
+ Truncate<String> trunc = Truncate.get(Types.StringType.get(), 7);
+ Expression expr = notStartsWith(COLUMN, "abcdefg");
+ BoundPredicate<String> boundExpr = (BoundPredicate<String>) Binder.bind(SCHEMA.asStruct(), expr, false);
+ UnboundPredicate<String> projected = trunc.project(COLUMN, boundExpr);
+ Evaluator evaluator = new Evaluator(SCHEMA.asStruct(), projected);
+
+ Assert.assertFalse("notStartsWith(abcdefg, truncate(abcdefg, 7)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcdefg")));
+
+ Assert.assertTrue("notStartsWith(abcdefg, truncate(ab, 2)) => true",
+ evaluator.eval(TestHelpers.Row.of("ab")));
+
+ Assert.assertTrue("notStartsWith(abcdefg, truncate(a, 16)) => true",
+ evaluator.eval(TestHelpers.Row.of("a")));
+ }
+
+ @Test
+ public void testStrictMetricsEvaluatorForNotStartsWith() {
+ boolean shouldRead = new StrictMetricsEvaluator(SCHEMA, notStartsWith(COLUMN, "bbb")).eval(FILE_1);
+ Assert.assertFalse("Should not match: strict metrics eval is always false for notStartsWith", shouldRead);
+ }
+
+ @Test
+ public void testInclusiveMetricsEvaluatorForNotStartsWith() {
Review comment:
Looks good.
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
+ private static final Schema SCHEMA = new Schema(FIELD);
+
+ // All 50 rows have someStringCol = 'bbb', none are null (despite being optional).
+ private static final DataFile FILE_1 = new TestDataFile("file_1.avro", Row.of(), 50,
+ // any value counts, including nulls
+ ImmutableMap.of(1, 50L),
+ // null value counts
+ ImmutableMap.of(1, 0L),
+ // nan value counts
+ null,
+ // lower bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")),
+ // upper bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")));
+
+ @Test
+ public void testTruncateProjections() {
+ PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).truncate(COLUMN, 4).build();
+
+ assertProjectionInclusive(spec, notStartsWith(COLUMN, "ab"), "ab", Expression.Operation.NOT_STARTS_WITH);
+ assertProjectionInclusive(spec, notStartsWith(COLUMN, "abab"), "abab", Expression.Operation.NOT_STARTS_WITH);
+ assertProjectionInclusive(spec, notStartsWith(COLUMN, "ababab"), "abab", Expression.Operation.NOT_STARTS_WITH);
+
+ assertProjectionStrict(spec, notStartsWith(COLUMN, "ab"), "ab", Expression.Operation.NOT_STARTS_WITH);
+ assertProjectionStrict(spec, notStartsWith(COLUMN, "abab"), "abab", Expression.Operation.NOT_EQ);
+
+ Expression projection = Projections.strict(spec).project(notStartsWith(COLUMN, "abcde"));
+ Assert.assertTrue(projection instanceof False);
+ }
+
+ @Test
+ public void testTruncateStringWhenProjectedPredicateTermIsLongerThanWidth() {
+ Truncate<String> trunc = Truncate.get(Types.StringType.get(), 2);
+ Expression expr = notStartsWith(COLUMN, "abcde");
+ BoundPredicate<String> boundExpr = (BoundPredicate<String>) Binder.bind(SCHEMA.asStruct(), expr, false);
+
+ UnboundPredicate<String> projected = trunc.project(COLUMN, boundExpr);
+ Evaluator evaluator = new Evaluator(SCHEMA.asStruct(), projected);
+
+ Assert.assertFalse("notStartsWith(abcde, truncate(abcde,2)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcde")));
+
+ Assert.assertFalse("notStartsWith(abcde, truncate(ab, 2)) => false",
+ evaluator.eval(TestHelpers.Row.of("ab")));
+
+ Assert.assertFalse("notStartsWith(abcde, truncate(abcdz, 2)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcdz")));
+
+ Assert.assertTrue("notStartsWith(abcde, truncate(a, 2)) => true",
+ evaluator.eval(TestHelpers.Row.of("a")));
+
+ Assert.assertTrue("notStartsWith(abcde, truncate(abzcde, 2)) => true",
+ evaluator.eval(TestHelpers.Row.of("azcde")));
+ }
+
+ @Test
+ public void testTruncateStringWhenProjectedPredicateTermIsShorterThanWidth() {
+ Truncate<String> trunc = Truncate.get(Types.StringType.get(), 16);
+ Expression expr = notStartsWith(COLUMN, "ab");
+ BoundPredicate<String> boundExpr = (BoundPredicate<String>) Binder.bind(SCHEMA.asStruct(), expr, false);
+ UnboundPredicate<String> projected = trunc.project(COLUMN, boundExpr);
+ Evaluator evaluator = new Evaluator(SCHEMA.asStruct(), projected);
+
+ Assert.assertFalse("notStartsWith(ab, truncate(abcde, 16)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcde")));
+
+ Assert.assertFalse("notStartsWith(ab, truncate(ab, 16)) => false",
+ evaluator.eval(TestHelpers.Row.of("ab")));
+
+ Assert.assertTrue("notStartsWith(ab, truncate(a, 16)) => true",
+ evaluator.eval(TestHelpers.Row.of("a")));
+ }
+
+ @Test
+ public void testTruncateStringWhenProjectedPredicateTermIsEqualToWidth() {
+ Truncate<String> trunc = Truncate.get(Types.StringType.get(), 7);
+ Expression expr = notStartsWith(COLUMN, "abcdefg");
+ BoundPredicate<String> boundExpr = (BoundPredicate<String>) Binder.bind(SCHEMA.asStruct(), expr, false);
+ UnboundPredicate<String> projected = trunc.project(COLUMN, boundExpr);
+ Evaluator evaluator = new Evaluator(SCHEMA.asStruct(), projected);
+
+ Assert.assertFalse("notStartsWith(abcdefg, truncate(abcdefg, 7)) => false",
+ evaluator.eval(TestHelpers.Row.of("abcdefg")));
+
+ Assert.assertTrue("notStartsWith(abcdefg, truncate(ab, 2)) => true",
+ evaluator.eval(TestHelpers.Row.of("ab")));
+
+ Assert.assertTrue("notStartsWith(abcdefg, truncate(a, 16)) => true",
+ evaluator.eval(TestHelpers.Row.of("a")));
+ }
+
+ @Test
+ public void testStrictMetricsEvaluatorForNotStartsWith() {
Review comment:
Looks good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567180036
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestSparkTableUtil.java
##########
@@ -397,6 +397,21 @@ public void testPartitionScanByFilter() {
Dataset<Row> partitionDF = SparkTableUtil.partitionDFByFilter(spark, QUALIFIED_TABLE_NAME, "data = 'a'");
Assert.assertEquals("There should be 1 matching partition", 1, partitionDF.count());
+
+ partitions = SparkTableUtil.getPartitionsByFilter(spark, QUALIFIED_TABLE_NAME, "data not like 'a'");
Review comment:
Probably didn't _need_ to update this, but it looks correct so let's keep them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r559263994
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/Spark3Util.java
##########
@@ -579,6 +579,8 @@ public String or(String leftResult, String rightResult) {
return pred.ref().name() + " != " + sqlString(pred.literal());
case STARTS_WITH:
return pred.ref().name() + " LIKE '" + pred.literal() + "%'";
+ case NOT_STARTS_WITH:
+ return pred.ref().name() + " NOT LIKE '" + pred.literal() + "%'";
Review comment:
Resolving this coversation as there is already a follow up issue https://github.com/apache/iceberg/issues/2059. I'll mention that `Spark3Util` shouuld be considered when working on that issue as a follow up.
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/Spark3Util.java
##########
@@ -579,6 +579,8 @@ public String or(String leftResult, String rightResult) {
return pred.ref().name() + " != " + sqlString(pred.literal());
case STARTS_WITH:
return pred.ref().name() + " LIKE '" + pred.literal() + "%'";
+ case NOT_STARTS_WITH:
+ return pred.ref().name() + " NOT LIKE '" + pred.literal() + "%'";
Review comment:
Resolving this conversation as there is already a follow up issue https://github.com/apache/iceberg/issues/2059. I'll mention that `Spark3Util` shouuld be considered when working on that issue as a follow up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772661753
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ // notStartsWith will match unless all values must start with the prefix. This happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
+ // if lower is shorter than the prefix, it can't start with the prefix
+ if (lower.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ int cmp = comparator.compare(BinaryUtil.truncateBinary(lower, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ ByteBuffer upper = upperBounds.get(id);
+ // if upper is shorter than the prefix, it can't start with the prefix
+ if (upper.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ cmp = comparator.compare(BinaryUtil.truncateBinary(upper, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ // both bounds match the prefix, so all rows must match the prefix and therefore do not satisfy
Review comment:
That makes sense as they've potentially been truncated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772689439
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java
##########
@@ -340,6 +340,56 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int pos = Accessors.toPosition(ref.accessor());
+ PartitionFieldSummary fieldStats = stats.get(pos);
+
+ // Iceberg does not implement SQL 3-boolean logic. Therefore, for all null values, we have decided to
+ // return ROWS_MIGHT_MATCH in order to allow the query engine to further evaluate this partition, as
+ // null does not start with any non-null value.
+ if (fieldStats.containsNull() || fieldStats.lowerBound() == null) {
Review comment:
Nevermind, this is correct as I mentioned above in the inclusive metrics evaluator. I'd probably change the comment and remove the part about 3-value logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772731055
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
I'd probably just remove the comment. It makes sense that a null value doesn't match a prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-998335347
Overall, the tests and implementation all look correct to me. I think there are a few minor things we could do, but I'm ready to commit this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772656008
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ // notStartsWith will match unless all values must start with the prefix. This happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
+ // if lower is shorter than the prefix, it can't start with the prefix
Review comment:
I can update it to that. I didn't actually write the comments for this one. They came directly as a suggestion of somebody else. Though I can see how `it` is confusing as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772719331
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -377,6 +381,31 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_CANNOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Boolean hasNonDictPage = isFallback.get(id);
+ if (hasNonDictPage == null || hasNonDictPage) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ Set<T> dictionary = dict(id, lit.comparator());
+
+ // Allow query engine to make its own decisions regarding SQL 3-valued boolean logic.
+ if (dictionary.contains(null)) {
+ return ROWS_MIGHT_MATCH;
+ }
Review comment:
The dictionary will never contain null, so you can remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555137226
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Expression.java
##########
@@ -74,6 +75,10 @@ public Operation negate() {
return Operation.NOT_IN;
case NOT_IN:
return Operation.IN;
+ case STARTS_WITH:
+ return Operation.NOT_STARTS_WITH;
Review comment:
I think you got these two flipped?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-985880725
> @cccs-eric, this is something we should complete. I didn't realize we hadn't finished it. @kbendick can you update this when you get a chance?
Yeah sure. I'd be happy to. I'll probably complete some of the work here and then open 1 or more smaller PRs that are newer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r762689770
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,27 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abc")));
Review comment:
Added. Passes as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567161493
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
Review comment:
Nit: indentation should be 2 indents / 4 spaces from the start of `if`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568264237
##########
File path: spark2/src/main/java/org/apache/iceberg/spark/SparkFilters.java
##########
@@ -135,6 +136,13 @@ public static Expression convert(Filter filter) {
case NOT:
Not notFilter = (Not) filter;
+ Filter sparkChild = notFilter.child();
+ // Special case to use the Iceberg NOT_STARTS_WITH operator for
+ // Not(StringStartsWith) to allow filter push down.
+ if (sparkChild instanceof StringStartsWith) {
+ StringStartsWith castedChild = (StringStartsWith) sparkChild;
+ return notStartsWith(castedChild.attribute(), castedChild.value());
+ }
Review comment:
Yeah I removed this and tested everything and it seems fine to go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554924932
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -69,6 +69,8 @@ public boolean test(T value) {
return cmp.compare(value, literal.value()) != 0;
case STARTS_WITH:
return String.valueOf(value).startsWith((String) literal.value());
+ case NOT_STARTS_WITH:
+ return !String.valueOf(value).startsWith((String) literal.value());
Review comment:
It's also tempting to just use the comparator from literal, like follows:
```java
cmp.compare(value, literal.value()) == 0;
```
However, because we're only interested in checking whether or not `value` does not start with the predicate's literal term, we would need to truncate like we do elsewhere so I don't think that the simple one line above would be correct (nice as it would be though).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567382673
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
Review comment:
Ok. I can definitely do that. A lot of the code style that I used was more or less chosen to mirror `startsWith` (so that both could be refactored together if need be).
I much prefer the commented version as well. I chose to leave it on the slimmer side with respect to comments as there are not necessarily a ton surrounding it, but I do agree that this makes it much easier to reason about (as well as ensure we avoid unnecessary allocations, etc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555380423
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -116,7 +116,11 @@ public R or(R leftResult, R rightResult) {
}
public <T> R startsWith(BoundReference<T> ref, Literal<T> lit) {
- throw new UnsupportedOperationException("Unsupported operation.");
+ throw new UnsupportedOperationException("startsWith operation is not supported by the visitor");
Review comment:
I updated all of the ones I mentioned above in this file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571701231
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
I've added a test on `all_nulls` here: https://github.com/apache/iceberg/pull/2062/files#diff-e18903de77ee699a6c58e3f785515ecd3f335f3fcf13a9de3c779c988f53f14bR431-R434
I'm going to rebase off of master momentarily and then I believe this test will need to be updated, but in essence it will stay the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383832
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Will do.
I just checked and you are correct, nulls are matched. I will add a test case for `all_nulls`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r559263109
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##########
@@ -151,5 +151,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
public <T> Boolean startsWith(Bound<T> valueExpr, Literal<T> lit) {
return ((String) valueExpr.eval(struct)).startsWith((String) lit.value());
}
+
+ @Override
+ public <T> Boolean notStartsWith(Bound<T> valueExpr, Literal<T> lit) {
+ return !((String) valueExpr.eval(struct)).startsWith((String) lit.value());
Review comment:
Update
For `ResidualEvaluator`, it can't be `!startsWith(ref, lit)` as `ResidualEvaluator` returns an expression, so it would have to be `startsWith(ref, lit).negate()`. This works since we're returning `alwaysTrue()` or `alwaysFalse()`, but it is a little confusing. Also, in the same file, we don't define `notIn` by using `in` in this way. I personally think `ResidualEvaluator` should be left as is.
Additionally, `BoundLiteralPredicate` cannot be defined in terms of `STARTS_WITH` as the function signature is just `test(T value)` and then the operation is caught in a switch statement and the operation is stored as a value on `this` (the `BoundLiteralPredicate` instance). So we'd have to return `negate().test(value)` which is also pretty confusing and likely error prone.
I'm going to close this conversation as it seems that the instances where we can easily and simply replace `notStartsWith` with `!starsWith` have all been covered. Feel free to reopen it or comment further if you disagree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r589250942
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -284,8 +284,30 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
if (predicate.isUnaryPredicate()) {
return Expressions.predicate(predicate.op(), name);
- } else if (predicate.isLiteralPredicate()) {
- return ProjectionUtil.truncateArray(name, predicate.asLiteralPredicate(), this);
+ } else if (predicate instanceof BoundLiteralPredicate) {
Review comment:
That's a good call. I'm not 100% sure why this got changed. If memory serves, I think I might have just used this snippet as it was recommended earlier in the review process (which I think might have had the `instanceof` check). So I chose to err on the side of caution and not deviate from whatever was provided.
Going forward, I will use my own judgement about whether there's a better way to write something and consider that what is suggested might potentially be a reference (particularly in this case where the existing code is already using `isLiteralPredicate` and the original recommendation has clear intent). I agree that keeping `isLiteralPredicate` function is for sure a better idea.
I will update this 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jun-he commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
jun-he commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r565083422
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
+ if (isStringPrefixPredicate(pred)) {
if (pred.literal().value().length() < width()) {
Review comment:
Seems missing `if pred.literal().value().length() > width()` case for `NOT_STARTS_WITH`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571701055
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
Review comment:
Please feel free to reopen this if there are any concerns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r762691292
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,27 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abc")));
+ Assert.assertFalse("Abc startsWith abc should be false", evaluator.eval(TestHelpers.Row.of("Abc")));
+ Assert.assertFalse("a startsWith abc should be false", evaluator.eval(TestHelpers.Row.of("a")));
+ Assert.assertTrue("abcd startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abcd")));
+ }
+
+ @Test
+ public void testNotStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, notStartsWith("s", "abc"));
+ Assert.assertFalse("abc notStartsWith abc should be false", evaluator.eval(TestHelpers.Row.of("abc")));
Review comment:
Good call. Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r557860590
##########
File path: api/src/main/java/org/apache/iceberg/expressions/StrictMetricsEvaluator.java
##########
@@ -402,6 +402,11 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_NOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ROWS_MIGHT_NOT_MATCH;
Review comment:
There's a TODO - `// TODO: Handle cases that definitely cannot match, such as notStartsWith("x") when the bounds are ["a", "b"].` - and once this is merged I'll be sure to open an issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cccs-eric commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator (including Spark 3.2)
Posted by GitBox <gi...@apache.org>.
cccs-eric commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-998377355
Thanks for the work @kbendick, will test it out once it is released!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772689015
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
Actually, this was my mistake.
Iceberg expressions implement normal logic and ignore 3-value logic. null does not start with any prefix. If there is a null value then it does not start with the given prefix and would match the expression. That means that `mayContainNull` should cause this to return `ROWS_MIGHT_MATCH` as you have here.
If SQL chooses to filter out those rows with 3-value logic, then that's okay. But Iceberg should return them because they match the Iceberg expression. I think what I was confused about was the mention of the 3-value logic here. I would probably remove that because the semantics of Iceberg expressions should not change, as is implied by the statement, "return ROWS_MIGHT_MATCH to allow the query engine to make its own decision". Instead, I'd change the comment to something like "null does not start with any prefix, any null row matches".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772660659
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ // notStartsWith will match unless all values must start with the prefix. This happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
+ // if lower is shorter than the prefix, it can't start with the prefix
Review comment:
I updated both the `it` for lower and for upper in this file. Will check the other evaluators for usage of `it` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772658767
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java
##########
@@ -340,6 +340,56 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int pos = Accessors.toPosition(ref.accessor());
+ PartitionFieldSummary fieldStats = stats.get(pos);
+
+ // Iceberg does not implement SQL 3-boolean logic. Therefore, for all null values, we have decided to
+ // return ROWS_MIGHT_MATCH in order to allow the query engine to further evaluate this partition, as
+ // null does not start with any non-null value.
+ if (fieldStats.containsNull() || fieldStats.lowerBound() == null) {
Review comment:
`fieldStats.lowerBound() == null` is checked in the `if` below, so no need to duplicate that here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r770855531
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ // notStartsWith will match unless all values must start with the prefix. This happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
+ // if lower is shorter than the prefix, it can't start with the prefix
+ if (lower.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ int cmp = comparator.compare(BinaryUtil.truncateBinary(lower, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ ByteBuffer upper = upperBounds.get(id);
+ // if upper is shorter than the prefix, it can't start with the prefix
+ if (upper.remaining() < prefixAsBytes.remaining()) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ cmp = comparator.compare(BinaryUtil.truncateBinary(upper, prefixAsBytes.remaining()), prefixAsBytes);
+ if (cmp == 0) {
+ // both bounds match the prefix, so all rows must match the prefix and therefore do not satisfy
Review comment:
nit: both bounds start with the prefix (?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567173568
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,24 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc => true", evaluator.eval(TestHelpers.Row.of("abc")));
+ Assert.assertFalse("abc startsWith Abc => false", evaluator.eval(TestHelpers.Row.of("Abc")));
+ Assert.assertFalse("a startsWith abc => false", evaluator.eval(TestHelpers.Row.of("a")));
+ }
+
+ @Test
+ public void testNotStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, notStartsWith("s", "abc"));
+ Assert.assertFalse("abc notStartsWith abc => false", evaluator.eval(TestHelpers.Row.of("abc")));
+ Assert.assertTrue("abc notStartsWith Abc => true", evaluator.eval(TestHelpers.Row.of("Abc")));
+ Assert.assertTrue("a notStartsWith abc => true", evaluator.eval(TestHelpers.Row.of("a")));
Review comment:
Could you also add a proper prefix case here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r554674753
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -91,6 +93,8 @@ public String toString() {
return term() + " != " + literal;
case STARTS_WITH:
return term() + " startsWith \"" + literal + "\"";
+ case NOT_STARTS_WITH:
+ return term() + " notStartsWith \"" + literal + "\"";
Review comment:
Again, I chose to go with the existing style of the code, but it does seem that the string representation of these predicates has too many quotes.
Here's an example: `ref(name="data") startsWith ""ab""`. I would expect that to be `ref(name="data") startsWith "ab"`.
It gets even weirder when using SQL, as I saw Spark showing `data NOT LIKE '"b"%'` in the explain output, though the predicate did properly evaluate to the expected `data NOT LIKE 'b%'`.
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java
##########
@@ -329,5 +329,36 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int pos = Accessors.toPosition(ref.accessor());
+ PartitionFieldSummary fieldStats = stats.get(pos);
+
+ // values are all null (and stats exist) and literal cannot contain null
+ if (fieldStats.lowerBound() == null) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ ByteBuffer lower = fieldStats.lowerBound();
+ // truncate lower bound so that its length in bytes is not greater than the length of prefix
+ int lowerLength = Math.min(prefixAsBytes.remaining(), lower.remaining());
+ int lowerCmp = comparator.compare(BinaryUtil.truncateBinary(lower, lowerLength), prefixAsBytes);
+ if (lowerCmp == 0) {
+ ByteBuffer upper = fieldStats.upperBound();
+ // truncate upper bound so that its length in bytes is not greater than the length of prefix
+ int upperLength = Math.min(prefixAsBytes.remaining(), upper.remaining());
+ int upperCmp = comparator.compare(BinaryUtil.truncateBinary(upper, upperLength), prefixAsBytes);
+ if (upperCmp == 0) {
+ return ROWS_CANNOT_MATCH;
+ }
+ }
Review comment:
I had originally made a utility function, `BinaryUtils.startsWith(ByteBuffer value, ByteBuffer prefix, Comparator<ByteBuffer> cmp)`, however there are a number of assumptions that are valid once we've gotten to the evaluators that don't necessarily hold in the general case, e.g. that the ByteBuffers are at `position` 0 (which should be the case here), so I decided to inline the definition the handful of times that it's needed as it's not too complicated.
If we think the utility function would be cleaner, I'd be happy to add it back in and also update STARTS_WITH to use it as well (it's inlined in the same manner, with different logic on the comparison result).
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveMetricsEvaluator.java
##########
@@ -529,6 +533,47 @@ public void testStringStartsWith() {
Assert.assertFalse("Should not read: range doesn't match", shouldRead);
}
+ @Test
+ // More interesting test cases are included in TestNotStartsWith
+ public void testStringNotStartsWith() {
+ boolean shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "a"), true).eval(FILE);
+ Assert.assertTrue("Should read: no stats", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "a"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aa"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aaa"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "1s"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "1str1x"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "ff"), true).eval(FILE_4);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "aB"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "dWX"), true).eval(FILE_2);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "5"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", "3str3x"), true).eval(FILE_3);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ String aboveMax = UnicodeUtil.truncateStringMax(Literal.of("イロハニホヘト"), 4).value().toString();
+ shouldRead = new InclusiveMetricsEvaluator(SCHEMA, notStartsWith("required", aboveMax), true).eval(FILE_4);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+ }
Review comment:
This is similar to the existing `testStringStartsWith` function above. However, the values with included metrics for the given files makes this test rather uninteresting.
I've seen elsewhere in the codebase where we have left the test function in with an empty body and junit's `@Ignore`, with a comment directing to where this is better tested - which is currently the newly added `TestNotStartsWith`, which has files where `notStartsWith` will not be a match. Possibly we should do that here?
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/Spark3Util.java
##########
@@ -579,6 +579,8 @@ public String or(String leftResult, String rightResult) {
return pred.ref().name() + " != " + sqlString(pred.literal());
case STARTS_WITH:
return pred.ref().name() + " LIKE '" + pred.literal() + "%'";
+ case NOT_STARTS_WITH:
+ return pred.ref().name() + " NOT LIKE '" + pred.literal() + "%'";
Review comment:
Here's another case where we possibly wind up with too many quotes. I can follow up in another PR. I chose to go with the existing format.
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -91,6 +93,8 @@ public String toString() {
return term() + " != " + literal;
case STARTS_WITH:
return term() + " startsWith \"" + literal + "\"";
+ case NOT_STARTS_WITH:
+ return term() + " notStartsWith \"" + literal + "\"";
Review comment:
I opened an issue for this as well, https://github.com/apache/iceberg/issues/2059. I can change both STARTS_WITH and NOT_STARTS_WITH if we think that there are in fact too many quotes in the string representation.
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestStartsWith.java
##########
@@ -67,8 +67,8 @@ public void testTruncateString() {
UnboundPredicate<String> projected = trunc.project(COLUMN, boundExpr);
Evaluator evaluator = new Evaluator(SCHEMA.asStruct(), projected);
- Assert.assertTrue("startsWith(abcde, truncate(abcde,2)) => true",
- evaluator.eval(TestHelpers.Row.of("abcde")));
+ Assert.assertTrue("startsWith(abcde, truncate(abcdg,2)) => true",
+ evaluator.eval(TestHelpers.Row.of("abcdg")));
Review comment:
I updated this test from abcde, as it seems much more revealing of a test that we're going to read a partition that actually does not start with the given value, because of the truncation width used in the partition spec.
I'd be happy to move this to anther PR given that it is relatively unrelated, or remove it if anybody feels that strongly about it,. It does seem to me to be much more informative of a test case than`abcde startsWith abcde`, as that would return true on an identity partition spec as well.
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
+ lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
+ ByteBuffer lower = lowerBounds.get(id);
Review comment:
I did not add a `null` check on lower, as I didn't see it in any of the other metrics based tests. But I thought I'd bring it up as this will definitely NPE if it's possible for the column to be in the `upperBounds` and `lowerBounds` map with a null value. Seems like that shouldn't happen though.
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -373,6 +373,25 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_CANNOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Boolean hasNonDictPage = isFallback.get(id);
+ if (hasNonDictPage == null || hasNonDictPage) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ Set<T> dictionary = dict(id, lit.comparator());
+ for (T item : dictionary) {
+ if (!item.toString().startsWith(lit.value().toString())) {
+ return ROWS_MIGHT_MATCH;
+ }
Review comment:
Here's one more case where we're using `.toString` and I wonder if we should be using one of the built in Literal Comparators instead.
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ResidualEvaluator.java
##########
@@ -214,6 +214,11 @@ public Expression or(Expression leftResult, Expression rightResult) {
return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysTrue() : alwaysFalse();
}
+ @Override
+ public <T> Expression notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ((String) ref.eval(struct)).startsWith((String) lit.value()) ? alwaysFalse() : alwaysTrue();
+ }
Review comment:
This is another case where I chose to go with the existing style of the surrounding code (e.g. the `startsWith` definition just above this), but I'm not so sure that casting to java String type is appropriate vs using the built in comparators.
Comparing the values as ByteBuffers, given the differences in Java's UTF-16 default and the included Literals comparator which handles UTF-8 code point differences for characters that are encoded as 3 vs 4 bytes, seems safer to me but I chose to be safe and emulate the surrounding code.
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestTruncate.java
##########
@@ -75,6 +75,8 @@ public void testTruncateString() {
"abcde", trunc.apply("abcdefg"));
Assert.assertEquals("Should not pad strings shorter than length",
"abc", trunc.apply("abc"));
+ Assert.assertEquals("Should not alter strings equal to length",
+ "abcde", trunc.apply("abcde"));
Review comment:
Not exactly related so happy to remove if anybody rejects (or move it into its own PR).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567179515
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
##########
@@ -500,6 +534,48 @@ public void testUnpartitionedStartsWith() {
Assert.assertEquals("junction", matchedData.get(0));
}
+ @Test
+ public void testUnpartitionedNotStartsWith() {
+ Dataset<Row> df = spark.read()
+ .format("iceberg")
+ .option(SparkReadOptions.VECTORIZATION_ENABLED, String.valueOf(vectorized))
+ .load(unpartitioned.toString());
+
+ List<String> matchedData = df.select("data")
+ .where("data NOT LIKE 'jun%'")
+ .as(Encoders.STRING())
+ .collectAsList();
+
+ List<String> expected = testRecords(SCHEMA).stream()
+ .map(r -> r.getField("data").toString())
+ .filter(d -> !d.startsWith("jun"))
+ .collect(Collectors.toList());
+
+ Assert.assertEquals(9, matchedData.size());
+ Assert.assertEquals(new HashSet<>(expected), new HashSet<>(matchedData));
Review comment:
This seems sufficient to test the Spark path without adding other tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383832
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Will do.
I had thought about the matching on `notStartsWith` for null boundaries, but I think I originally did not check properly. I just checked and nulls are matched when using Spark.
```
scala> val df = Seq("hi", null, "there", "world").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
df.selectExpr("word not like 'wo%' as word").show(false)
+-----+
|word |
+-----+
|true |
|null |
|true |
|false|
+-----+
scala> df.select(not($"word".startsWith("wo"))).show
+--------------------------+
|(NOT startswith(word, wo))|
+--------------------------+
| true|
| null|
| true|
| false|
+--------------------------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r569816358
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java
##########
@@ -329,5 +329,36 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int pos = Accessors.toPosition(ref.accessor());
+ PartitionFieldSummary fieldStats = stats.get(pos);
+
+ // values are all null (and stats exist) and literal cannot contain null
+ if (fieldStats.lowerBound() == null) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ ByteBuffer lower = fieldStats.lowerBound();
+ // truncate lower bound so that its length in bytes is not greater than the length of prefix
+ int lowerLength = Math.min(prefixAsBytes.remaining(), lower.remaining());
+ int lowerCmp = comparator.compare(BinaryUtil.truncateBinary(lower, lowerLength), prefixAsBytes);
+ if (lowerCmp == 0) {
+ ByteBuffer upper = fieldStats.upperBound();
+ // truncate upper bound so that its length in bytes is not greater than the length of prefix
+ int upperLength = Math.min(prefixAsBytes.remaining(), upper.remaining());
+ int upperCmp = comparator.compare(BinaryUtil.truncateBinary(upper, upperLength), prefixAsBytes);
+ if (upperCmp == 0) {
+ return ROWS_CANNOT_MATCH;
+ }
+ }
Review comment:
I have made the same changes as in `InclusiveMericsEvaluator`.
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
Review comment:
I've updated to this definition.
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
I had some concerns about my original Spark based check, so I checked again. I was mistaken when I initially checked to see what Spark returns, as Spark implements SQL 3-way boolean logic, for example:
```
scala> val df = Seq("aa", "bb", null, "b").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
scala> df.createOrReplaceTempView("t")
scala> df.select(not($"word".startsWith("bb"))).show
+--------------------------+
|(NOT startswith(word, bb))|
+--------------------------+
| true|
| false|
| null|
| true|
+--------------------------+
```
However, Iceberg does not implement 3-way boolean logic. So we need to make a decision on whether or not we want Iceberg to return `true` or `false` in this case.
I discussed this offline with @rdblue and we've come up with the following rationale for an `all_nulls` file matching a `notStartsWith` filter:
The argument in favor of returning `true` is stronger than deciding to return `false`, i.e. by saying that null does not start with "bb" or whatever the predicate's value is. The reason for this is that:
(1) Logically, null does not start with "bb" (or any non-null value)
(2) If we have to choose between Iceberg surfacing too many values or too few, we would want Iceberg to return more values and then allow the query engine to make the final decision on the null value. If we return too few values, the query engine might not return the results that it would have if the underlying table format were not Iceberg. However, if we return too many values from the predicate filters, as in this case `true` means maybe and so the query engine can still ensure that the result is correct (according to the query engines definition of correct). When `false` means definitely not, the query engine has no ability to catch anything.
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestPredicateBinding.java
##########
@@ -108,6 +110,25 @@ public void testComparisonPredicateBinding() {
}
}
+ @Test
+ @SuppressWarnings("unchecked")
+ public void tesPredicateBindingForStringPrefixComparisons() {
Review comment:
Good eyes. I've fixed this typo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383832
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Will do.
I just checked and you are correct, nulls are matched (when using Spark below). I will add a test case for `all_nulls`.
```
scala> val df = Seq("hi", null, "there", "world").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
df.selectExpr("word not like 'wo%' as word").show(false)
+-----+
|word |
+-----+
|true |
|null |
|true |
|false|
+-----+
scala> df.select(not($"word".startsWith("wo"))).show
+--------------------------+
|(NOT startswith(word, wo))|
+--------------------------+
| true|
| null|
| true|
| false|
+--------------------------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-759143183
I think a timeout or something caused the Java 8 tests to fail. Running them locally, they all pass (and the Java 11 tests pass as well).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick edited a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick edited a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-782361915
Looking into what's been done since I opened this, Flink now has filter pushdown. So we likely need to update `org.apache.iceberg.flink.FlinkFilters` to account for NOT_STARTS_WITH. Or at the least we might consider adding some tests in Flink that use `NOT LIKE` to ensure those queries will work correctly.
I will look into that asap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555135021
##########
File path: api/src/main/java/org/apache/iceberg/expressions/BoundLiteralPredicate.java
##########
@@ -69,6 +69,8 @@ public boolean test(T value) {
return cmp.compare(value, literal.value()) != 0;
case STARTS_WITH:
return String.valueOf(value).startsWith((String) literal.value());
+ case NOT_STARTS_WITH:
+ return !String.valueOf(value).startsWith((String) literal.value());
Review comment:
I think alignment with the previous code is find. I would be worried about any changes to past matching behavior and I think one of the concerns here is that there are multiple groups of bytes which may represent the same string so you can't just compare the binary values I think....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567164331
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -372,6 +372,38 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ if (containsNullsOnly(id)) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ if (lowerBounds != null && upperBounds != null &&
Review comment:
I think this could definitely use a comment. It took me a while to work my way through it.
If I understand correctly, we can return `ROWS_CANNOT_MATCH` when we can guarantee that all rows must start with the prefix. That happens when the lower bound starts with the prefix and the upper bound starts with the prefix.
Also, we truncate the lower and upper bounds at the length of the prefix. If the bound is shorter than the prefix, then we know it does not start with the prefix. I think it is a bit confusing that this case is not directly handled because you have to think it through. That also might create a copy of the buffer to do the compare when we don't actually need to. To make it more clear and avoid any unnecessary copies, I'd do this:
```java
// notStartsWith will match unless all values must start with the prefix. this happens when the lower and upper
// bounds both start with the prefix.
if (lowerBounds != null && upperBounds != null &&
lowerBounds.containsKey(id) && upperBounds.containsKey(id)) {
ByteBuffer lower = lowerBounds.get(id);
// if lower is shorter than the prefix, it can't start with the prefix
if (lower.remaining() < prefixAsBytes.remaining()) {
return ROWS_MIGHT_MATCH;
}
// truncate lower bound to the prefix and check for equality
int cmp = comparator.compare(BinaryUtil.truncateBinary(lower, prefixAsBytes.remaining()), prefixAsBytes);
if (cmp == 0) {
// the lower bound starts with the prefix; check the upper bound
ByteBuffer upper = upperBounds.get(id);
// if upper is shorter than the prefix, it can't start with the prefix
if (upper.remaining() < prefixAsBytes.remaining()) {
return ROWS_MIGHT_MATCH;
}
// truncate upper bound so that its length in bytes is not greater than the length of prefix
cmp = comparator.compare(BinaryUtil.truncateBinary(upper, prefixAsBytes.remaining()), prefixAsBytes);
if (cmp == 0) {
// both bounds match the prefix, so all rows must match the prefix and none do not match
return ROWS_CANNOT_MATCH;
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555139230
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -116,7 +116,11 @@ public R or(R leftResult, R rightResult) {
}
public <T> R startsWith(BoundReference<T> ref, Literal<T> lit) {
- throw new UnsupportedOperationException("Unsupported operation.");
+ throw new UnsupportedOperationException("startsWith operation is not supported by the visitor");
Review comment:
nit: startsWith expression
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-792586478
Thank you so much for the further reviews everyone.
I looked into the Flink side of predicate push down, and if memory serves me there are no changes required. At the moment, I am occupied with some work-related things, but I will return to this PR asap. I spoke to @rdblue about it already as he has most recently reviewed but I've been fortunate to have several reviewers so I wanted to update you.
I will address the feedback and make any necessary updates / rebase to current master asap. I should hopefully be able to return to this PR pretty soon but as I mentioned I have to deal with some work things. Thanks for your understanding. Please reach out if this becomes blocking or more urgent for anybody's needs and I will do my best to accommodate that :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571696892
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
+ private static final Schema SCHEMA = new Schema(FIELD);
+
+ // All 50 rows have someStringCol = 'bbb', none are null (despite being optional).
+ private static final DataFile FILE_1 = new TestDataFile("file_1.avro", Row.of(), 50,
+ // any value counts, including nulls
+ ImmutableMap.of(1, 50L),
+ // null value counts
+ ImmutableMap.of(1, 0L),
+ // nan value counts
+ null,
+ // lower bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")),
+ // upper bounds
+ ImmutableMap.of(1, toByteBuffer(StringType.get(), "bbb")));
+
+ @Test
+ public void testTruncateProjections() {
Review comment:
You are correct. I am pushing a fix now that updates all of the projection tests, as well as adds assertions that the truncation transform is being respected. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567164743
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java
##########
@@ -329,5 +329,36 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int pos = Accessors.toPosition(ref.accessor());
+ PartitionFieldSummary fieldStats = stats.get(pos);
+
+ // values are all null (and stats exist) and literal cannot contain null
+ if (fieldStats.lowerBound() == null) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ ByteBuffer lower = fieldStats.lowerBound();
+ // truncate lower bound so that its length in bytes is not greater than the length of prefix
+ int lowerLength = Math.min(prefixAsBytes.remaining(), lower.remaining());
+ int lowerCmp = comparator.compare(BinaryUtil.truncateBinary(lower, lowerLength), prefixAsBytes);
+ if (lowerCmp == 0) {
+ ByteBuffer upper = fieldStats.upperBound();
+ // truncate upper bound so that its length in bytes is not greater than the length of prefix
+ int upperLength = Math.min(prefixAsBytes.remaining(), upper.remaining());
+ int upperCmp = comparator.compare(BinaryUtil.truncateBinary(upper, upperLength), prefixAsBytes);
+ if (upperCmp == 0) {
+ return ROWS_CANNOT_MATCH;
+ }
+ }
Review comment:
Sounds good to me. Could you make the same changes as in the `InclusiveMetricsEvaluator`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383832
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Will do.
I just checked and you are correct, nulls are matched (when using Spark below). I will add a test case for `all_nulls`.
```
scala> val df = Seq("hi", null, "there", "world").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
df.selectExpr("word not like 'wo%' as word").show
+-----+
|word |
+-----+
|true |
|null |
|true |
|false|
+-----+
scala> df.select(not($"word".startsWith("wo"))).show
+--------------------------+
|(NOT startswith(word, wo))|
+--------------------------+
| true|
| null|
| true|
| false|
+--------------------------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567175553
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
Review comment:
This field is only used on the next line. Maybe we don't need to create a constant?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567177688
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java
##########
@@ -472,6 +472,50 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Long valueCount = valueCounts.get(id);
+ if (valueCount == null) {
+ // the column is not present and is all nulls
+ return ROWS_CANNOT_MATCH;
+ }
+
+ Statistics<Binary> colStats = (Statistics<Binary>) stats.get(id);
+ if (colStats != null && !colStats.isEmpty()) {
+ if (hasNonNullButNoMinMax(colStats, valueCount)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ if (!colStats.hasNonNullValue()) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ Binary lower = colStats.genericGetMin();
+ // truncate lower bound so that its length in bytes is not greater than the length of prefix
+ int lowerLength = Math.min(prefixAsBytes.remaining(), lower.length());
+ int lowerCmp = comparator.compare(BinaryUtil.truncateBinary(lower.toByteBuffer(), lowerLength), prefixAsBytes);
+ if (lowerCmp == 0) {
+ Binary upper = colStats.genericGetMax();
+ // truncate upper bound so that its length in bytes is not greater than the length of prefix
+ int upperLength = Math.min(prefixAsBytes.remaining(), upper.length());
+ ByteBuffer upperByteBuffer = upper.toByteBuffer();
+ int upperCmp = comparator.compare(BinaryUtil.truncateBinary(upperByteBuffer, upperLength), prefixAsBytes);
+ if (upperCmp == 0) {
+ return ROWS_CANNOT_MATCH;
+ }
+ }
+ }
Review comment:
I think the logic should be updated here as well to what I suggested above for the lower = upper = prefix case.
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java
##########
@@ -472,6 +472,50 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Long valueCount = valueCounts.get(id);
+ if (valueCount == null) {
+ // the column is not present and is all nulls
+ return ROWS_CANNOT_MATCH;
+ }
+
+ Statistics<Binary> colStats = (Statistics<Binary>) stats.get(id);
+ if (colStats != null && !colStats.isEmpty()) {
+ if (hasNonNullButNoMinMax(colStats, valueCount)) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ if (!colStats.hasNonNullValue()) {
+ return ROWS_CANNOT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ Binary lower = colStats.genericGetMin();
+ // truncate lower bound so that its length in bytes is not greater than the length of prefix
+ int lowerLength = Math.min(prefixAsBytes.remaining(), lower.length());
+ int lowerCmp = comparator.compare(BinaryUtil.truncateBinary(lower.toByteBuffer(), lowerLength), prefixAsBytes);
+ if (lowerCmp == 0) {
+ Binary upper = colStats.genericGetMax();
+ // truncate upper bound so that its length in bytes is not greater than the length of prefix
+ int upperLength = Math.min(prefixAsBytes.remaining(), upper.length());
+ ByteBuffer upperByteBuffer = upper.toByteBuffer();
+ int upperCmp = comparator.compare(BinaryUtil.truncateBinary(upperByteBuffer, upperLength), prefixAsBytes);
+ if (upperCmp == 0) {
+ return ROWS_CANNOT_MATCH;
+ }
+ }
+ }
Review comment:
I think the logic should be updated here as well to what I suggested above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568177919
##########
File path: api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java
##########
@@ -197,6 +197,8 @@ private ProjectionUtil() {
return predicate(Expression.Operation.EQ, name, transform.apply(boundary));
case STARTS_WITH:
return predicate(Expression.Operation.STARTS_WITH, name, transform.apply(boundary));
+ case NOT_STARTS_WITH:
+ return predicate(Expression.Operation.NOT_STARTS_WITH, name, transform.apply(boundary));
Review comment:
Yeah, I think this should be removed and handled by the default case, which will return null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] cccs-eric commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
cccs-eric commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r762639059
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,27 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abc")));
Review comment:
Should we add a test with the pattern present, but not a the start?
```suggestion
Assert.assertTrue("abc startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abc")));
Assert.assertFalse("xabc startsWith abc should be false", evaluator.eval(TestHelpers.Row.of("xabc")));
```
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,27 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abc")));
+ Assert.assertFalse("Abc startsWith abc should be false", evaluator.eval(TestHelpers.Row.of("Abc")));
+ Assert.assertFalse("a startsWith abc should be false", evaluator.eval(TestHelpers.Row.of("a")));
+ Assert.assertTrue("abcd startsWith abc should be true", evaluator.eval(TestHelpers.Row.of("abcd")));
+ }
+
+ @Test
+ public void testNotStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, notStartsWith("s", "abc"));
+ Assert.assertFalse("abc notStartsWith abc should be false", evaluator.eval(TestHelpers.Row.of("abc")));
Review comment:
Should we add a test with the pattern present, but not a the start?
```suggestion
Assert.assertFalse("abc notStartsWith abc should be false", evaluator.eval(TestHelpers.Row.of("abc")));
Assert.assertTrue("xabc notStartsWith abc should be true", evaluator.eval(TestHelpers.Row.of("xabc")));
```
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -242,6 +248,10 @@ public R or(R leftResult, R rightResult) {
throw new UnsupportedOperationException("Unsupported operation.");
}
+ public <T> R notStartsWith(Bound<T> expr, Literal<T> lit) {
+ throw new UnsupportedOperationException("Unsupported operation.");
Review comment:
I think the exception message should specify which operation is not supported. `startsWith` should also be modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r762686988
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##########
@@ -242,6 +248,10 @@ public R or(R leftResult, R rightResult) {
throw new UnsupportedOperationException("Unsupported operation.");
}
+ public <T> R notStartsWith(Bound<T> expr, Literal<T> lit) {
+ throw new UnsupportedOperationException("Unsupported operation.");
Review comment:
I think generally we try to touch as few lines as needed in some cases (or avoid unnecessary changes), as that makes it easier on people who maintain forks.
I don't disagree with you that the error message should be more descriptive, but given what I know about the API, I don't think these specific base class for `BoundVisitor` actually get called anywhere. Notice how many of them return `null` as well.
As for the specific error message, it's just to be consistent with the existing one for `startsWith`.
I think given the size of the PR already, it's best to keep unrelated changes out and in a separate PR (particularly updating `startsWith` but until then, let's be consistent).
Large PRs like this already face a hurdle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772719123
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -368,6 +368,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
}
Set<T> dictionary = dict(id, lit.comparator());
+ if (dictionary.contains(null)) {
Review comment:
@kbendick, I don't think that this is a valid case. Parquet never encodes null values in its dictionaries. We can remove this. That avoids the need for the comment, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772659986
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java
##########
@@ -340,6 +340,56 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int pos = Accessors.toPosition(ref.accessor());
+ PartitionFieldSummary fieldStats = stats.get(pos);
+
+ // Iceberg does not implement SQL 3-boolean logic. Therefore, for all null values, we have decided to
+ // return ROWS_MIGHT_MATCH in order to allow the query engine to further evaluate this partition, as
+ // null does not start with any non-null value.
+ if (fieldStats.containsNull() || fieldStats.lowerBound() == null) {
Review comment:
I don't think that we need the null check here. Null values do not match, but we can't tell whether _all_ the values are null or not from these stats. So all we can do is ignore this and check the bounds.
You should be able to just remove this `if` block entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772661747
##########
File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java
##########
@@ -408,6 +408,54 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ Integer id = ref.fieldId();
+
+ // Iceberg does not implement SQL 3-boolean logic. Return ROWS_MIGHT_MATCH
+ // to allow the query engine to make its own decision regarding NULL values.
+ if (mayContainNull(id)) {
+ return ROWS_MIGHT_MATCH;
Review comment:
I'm not sure what I was thinking at the time, but I'm pretty sure it was wrong! No need to mention SQL boolean logic. These classes implement 2-valued logic correctly and we handle SQL translation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] tinder-kbendickson commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
tinder-kbendickson commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r566554091
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
+ if (isStringPrefixPredicate(pred)) {
if (pred.literal().value().length() < width()) {
Review comment:
So I spent some time analyzing this. There never was a case for that here with STARTS_WITH either.
In the case that `pred.literal().value().length() > width()`, it will hop out of this block and then hit a `return null` towards the very bottom.
I personally feel it's a bit confusing, but was hesistant to change it given the large scope of this PR and the fact that it would change existing code behavior (or we'd be implying that this was for sure the correct thing to do if we explicitly `return null` in the case that length > width() - which I didn't want to make that kind of implication in this PR).
I'm happy to update it, but we should be noting that we'll be changing the existing behavior of STARTS_WITH (unless that block explicitly returns null).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick removed a comment on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick removed a comment on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-759143183
I think a timeout or something caused the Java 8 tests to fail. Running them locally, they all pass (and the Java 11 tests pass as well).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555557019
##########
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##########
@@ -151,5 +151,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
public <T> Boolean startsWith(Bound<T> valueExpr, Literal<T> lit) {
return ((String) valueExpr.eval(struct)).startsWith((String) lit.value());
}
+
+ @Override
+ public <T> Boolean notStartsWith(Bound<T> valueExpr, Literal<T> lit) {
+ return !((String) valueExpr.eval(struct)).startsWith((String) lit.value());
Review comment:
I've fixed the usage in this evaluator, please let me know if you think I should do the same in `ResidualEvaluator` and `BoundLiteralPredicate`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r609320484
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
I've added a test for `all_nulls_mising_nan` (which is a string column) which passes: https://github.com/apache/iceberg/pull/2062/files#diff-e18903de77ee699a6c58e3f785515ecd3f335f3fcf13a9de3c779c988f53f14bR136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567175640
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
+ private static final Schema SCHEMA = new Schema(FIELD);
+
+ // All 50 rows have someStringCol = 'bbb', none are null (despite being optional).
+ private static final DataFile FILE_1 = new TestDataFile("file_1.avro", Row.of(), 50,
+ // any value counts, including nulls
Review comment:
Nit: indentation is off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568249208
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
Review comment:
Removed 👍
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
+ private static final Schema SCHEMA = new Schema(FIELD);
+
+ // All 50 rows have someStringCol = 'bbb', none are null (despite being optional).
+ private static final DataFile FILE_1 = new TestDataFile("file_1.avro", Row.of(), 50,
+ // any value counts, including nulls
Review comment:
I moved the indentation further to be aligned with the `a` in `private` (which is how I saw it in another test file). My iceberg checkstyle that I ran in intellij didn't complain, so please let me know if the indentation should be different. 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567383832
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
Will do.
I just checked (using Spark) and you are correct, nulls should be matched. I will add a test case for `all_nulls`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567170424
##########
File path: api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java
##########
@@ -197,6 +197,8 @@ private ProjectionUtil() {
return predicate(Expression.Operation.EQ, name, transform.apply(boundary));
case STARTS_WITH:
return predicate(Expression.Operation.STARTS_WITH, name, transform.apply(boundary));
+ case NOT_STARTS_WITH:
+ return predicate(Expression.Operation.NOT_STARTS_WITH, name, transform.apply(boundary));
Review comment:
Actually, from looking at the handling in `Truncate`, I think we can handle this without using the `truncateArray` methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r571705583
##########
File path: spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetReadMetadataColumns.java
##########
@@ -77,6 +77,9 @@
private static final int NUM_ROWS = 1000;
private static final List<InternalRow> DATA_ROWS;
private static final List<InternalRow> EXPECTED_ROWS;
+ // We filter for $"data" notStartsWith str1, along the row group splits, to ensure the row group
Review comment:
So the reason I originally put this in this file is because there are some tests along the splits.
What I'm looking to test here is that the filter is properly applied to row groups - by checking that the spark parquet reader returns all rows from a given split if any row in that split matches the filter.
I'm going to move this test to its own file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r569859335
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestInclusiveManifestEvaluator.java
##########
@@ -389,6 +396,39 @@ public void testStringStartsWith() {
Assert.assertFalse("Should skip: range doesn't match", shouldRead);
}
+ @Test
+ public void testStringNotStartsWith() {
+ boolean shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "aa"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "dddd"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "z"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("no_nulls", "a"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "zzzz"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
+
+ shouldRead = ManifestEvaluator.forRowFilter(notStartsWith("some_nulls", "1"), SPEC, false).eval(FILE);
+ Assert.assertTrue("Should read: range matches", shouldRead);
Review comment:
I had some concerns about my original Spark based check, so I checked again. I was mistaken when I initially checked to see what Spark returns, as Spark implements SQL 3-way boolean logic, for example:
```
scala> val df = Seq("aa", "bb", null, "b").toDF("word")
df: org.apache.spark.sql.DataFrame = [word: string]
scala> df.createOrReplaceTempView("t")
scala> df.select(not($"word".startsWith("bb"))).show
+--------------------------+
|(NOT startswith(word, bb))|
+--------------------------+
| true|
| false|
| null|
| true|
+--------------------------+
scala> spark.sql("SELECT * FROM t WHERE word NOT LIKE 'bb%'").show
+----+
|word|
+----+
| aa|
| b|
+----+
```
However, Iceberg does not implement 3-way boolean logic. So we need to make a decision on whether or not we want Iceberg to return `true` or `false` in this case.
I discussed this offline with @rdblue and we've come up with the following rationale for an `all_nulls` file matching a `notStartsWith` filter:
The argument in favor of returning `true` is stronger than deciding to return `false`, i.e. by saying that null does not start with "bb" or whatever the predicate's value is. The reason for this is that:
(1) Logically, null does not start with "bb" (or any non-null value)
(2) If we have to choose between Iceberg surfacing too many values or too few, we would want Iceberg to return more values and then allow the query engine to make the final decision on the null value. If we return too few values, the query engine might not return the results that it would have if the underlying table format were not Iceberg. However, if we return too many values from the predicate filters (as in this case - where `true` means maybe), the query engine can still ensure that the result is correct (according to the query engines definition of correct). When `false` means definitely not, the query engine has no ability to catch anything.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567171867
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
+ if (isStringPrefixPredicate(pred)) {
if (pred.literal().value().length() < width()) {
Review comment:
For the strict case, as I noted in a comment above, the predicate produced by truncating the value actually does result in a more strict match. If `notStartsWith("abc")` is strict projected with `truncate[2]`, it produces `notStartsWith("ab")`. And if something doesn't start with "ab" then it can't start with "abc". So we can add that case here.
This is not true for `startsWith` because truncating makes `startsWith` match more strings. I recommend changing this logic:
```
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
switch (pred.op()) {
case STARTS_WITH:
if (pred.literal().value().length() < width()) {
return Expressions.predicate(pred.op(), name, pred.literal().value());
} else if (pred.literal().value().length() == width()) {
return Expressions.equal(name, pred.literal().value());
}
return null;
case NOT_STARTS_WITH:
if (pred.literal().value().length() < width()) {
return Expressions.predicate(pred.op(), name, pred.literal().value());
} else if (pred.literal().value().length() == width()) {
return Expressions.notEqual(name, pred.literal().value());
}
return Expressions.predicate(pred.op(), name, apply(pred.literal().value()).toString());
default:
return ProjectionUtil.truncateArrayStrict(name, pred, this);
}
} else if (predicate.isSetPredicate() && predicate.op() == Expression.Operation.NOT_IN) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555363288
##########
File path: api/src/main/java/org/apache/iceberg/expressions/StrictMetricsEvaluator.java
##########
@@ -402,6 +402,11 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_NOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ROWS_MIGHT_NOT_MATCH;
Review comment:
Cool. I had a note in my working branch about possible cases that could be handled, but I figured this PR is already large and so I figured that `ROWS_MIGHT_NOT_MATCH` will definitely give correct results and matches the surrounding code.
I'lll leave this comment up and open a follow up issue after addressing some of the comments here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568249208
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
Review comment:
Removed 👍
##########
File path: api/src/test/java/org/apache/iceberg/transforms/TestNotStartsWith.java
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.transforms;
+
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.TestHelpers;
+import org.apache.iceberg.TestHelpers.Row;
+import org.apache.iceberg.TestHelpers.TestDataFile;
+import org.apache.iceberg.expressions.Binder;
+import org.apache.iceberg.expressions.BoundPredicate;
+import org.apache.iceberg.expressions.Evaluator;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.False;
+import org.apache.iceberg.expressions.InclusiveMetricsEvaluator;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.expressions.Projections;
+import org.apache.iceberg.expressions.StrictMetricsEvaluator;
+import org.apache.iceberg.expressions.UnboundPredicate;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.types.Types.NestedField;
+import org.apache.iceberg.types.Types.StringType;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static org.apache.iceberg.TestHelpers.assertAndUnwrapUnbound;
+import static org.apache.iceberg.expressions.Expressions.notStartsWith;
+import static org.apache.iceberg.types.Conversions.toByteBuffer;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+
+public class TestNotStartsWith {
+
+ private static final String COLUMN = "someStringCol";
+ private static final NestedField FIELD = optional(1, COLUMN, Types.StringType.get());
+ private static final Schema SCHEMA = new Schema(FIELD);
+
+ // All 50 rows have someStringCol = 'bbb', none are null (despite being optional).
+ private static final DataFile FILE_1 = new TestDataFile("file_1.avro", Row.of(), 50,
+ // any value counts, including nulls
Review comment:
I moved the indentation further to be aligned with the `a` in `private` (which is how I saw it in another test file). My iceberg checkstyle that I ran in intellij didn't complain, so please let me know if the indentation should be different. 👍
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestFilteredScan.java
##########
@@ -469,6 +473,21 @@ public void testPartitionedByDataStartsWithFilter() {
Assert.assertEquals(1, reader.planInputPartitions().size());
}
+ @Test
+ public void testPartitionedByDataNotStartsWithFilter() {
+ File location = buildPartitionedTable("partitioned_by_data", PARTITION_BY_DATA, "data_ident", "data");
+
+ DataSourceOptions options = new DataSourceOptions(ImmutableMap.of(
+ "path", location.toString())
+ );
Review comment:
Agreed.
##########
File path: core/src/test/java/org/apache/iceberg/TestManifestReaderStats.java
##########
@@ -82,6 +88,13 @@ public void testReadWithFilterIncludesFullStats() throws IOException {
ManifestEntry<DataFile> entry = entries.iterator().next();
assertFullStats(entry.file());
}
+
+ try (ManifestReader<DataFile> reader = ManifestFiles.read(manifest, FILE_IO)
+ .filterRows(Expressions.notStartsWith("data", "Z"))) {
+ CloseableIterable<ManifestEntry<DataFile>> entries = reader.entries();
+ ManifestEntry<DataFile> entry = entries.iterator().next();
+ assertFullStats(entry.file());
+ }
Review comment:
You're right. I think this is one of the cases where I added in several tests in order to better understand how everything was executed (occasionally hopping through with the debugger). I'll remove the changes to this file.
##########
File path: spark2/src/main/java/org/apache/iceberg/spark/SparkFilters.java
##########
@@ -135,6 +136,13 @@ public static Expression convert(Filter filter) {
case NOT:
Not notFilter = (Not) filter;
+ Filter sparkChild = notFilter.child();
+ // Special case to use the Iceberg NOT_STARTS_WITH operator for
+ // Not(StringStartsWith) to allow filter push down.
+ if (sparkChild instanceof StringStartsWith) {
+ StringStartsWith castedChild = (StringStartsWith) sparkChild;
+ return notStartsWith(castedChild.attribute(), castedChild.value());
+ }
Review comment:
Yeah I removed this and tested everything and it seems fine to go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r555565045
##########
File path: api/src/main/java/org/apache/iceberg/expressions/StrictMetricsEvaluator.java
##########
@@ -402,6 +402,11 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_NOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ return ROWS_MIGHT_NOT_MATCH;
Review comment:
I added a TODO in the code, similar to some of the other TODO's I've seen. Once this is merged I'll open an issue for a follow-up. Figure it's best not to open the issue until the issue exists in the master branch. 🙂
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r568071302
##########
File path: api/src/test/java/org/apache/iceberg/expressions/TestEvaluator.java
##########
@@ -220,6 +222,24 @@ public void testNotEqual() {
}
+ @Test
+ public void testStartsWith() {
+ StructType struct = StructType.of(required(24, "s", Types.StringType.get()));
+ Evaluator evaluator = new Evaluator(struct, startsWith("s", "abc"));
+ Assert.assertTrue("abc startsWith abc => true", evaluator.eval(TestHelpers.Row.of("abc")));
+ Assert.assertFalse("abc startsWith Abc => false", evaluator.eval(TestHelpers.Row.of("Abc")));
+ Assert.assertFalse("a startsWith abc => false", evaluator.eval(TestHelpers.Row.of("a")));
Review comment:
I can add that. With respect to the syntax, I copied a similar variant to what is found elsewhere. For converting it to typical "should" test syntax, would `abcd startsWith abc should be true` work?
The original syntax for this which I modified somewhat comes from a test in `TestStartsWith` (and possibly elsewhere), which reads as `"startsWith(abcde, truncate(abcdg,2)) => true"` (where the test row is "abcdg"). For me, I have to think about what exactly that statement is expressing a little harder than I'd like, but I will admit I haven't had all of my morning coffee so maybe it's plenty clear expressed that way?
I'm open to suggestions, but I'll switch to `abcd startsWith abc should be true` for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-876045720
Rebased this onto master so that we can come back to it if desired (as @holdenk has some on going python work for `starts_with`).
If we move forward with this, I'm happy to rebase / squash. 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] tinder-kbendickson commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
tinder-kbendickson commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r566554091
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##########
@@ -303,11 +303,15 @@ public boolean satisfiesOrderOf(Transform<?, ?> other) {
return Expressions.predicate(predicate.op(), name);
} else if (predicate instanceof BoundLiteralPredicate) {
BoundLiteralPredicate<CharSequence> pred = predicate.asLiteralPredicate();
- if (pred.op() == Expression.Operation.STARTS_WITH) {
+ if (isStringPrefixPredicate(pred)) {
if (pred.literal().value().length() < width()) {
Review comment:
So I spent some time analyzing this. There never was a case for that here with STARTS_WITH either.
In the case that `pred.literal().value().length() > width()`, it will hop out of this block and then hit a `return null` towards the very bottom.
I personally feel it's a bit confusing, but was hesistant to change it given the large scope of this PR and the fact that it would change existing code behavior (or we'd be implying that this was for sure the correct thing to do if we explicitly `return null` in the case that length > width() - which I didn't want to make that kind of implication in this PR).
I'm happy to update it, but we should be noting that we'll be changing the existing behavior of STARTS_WITH (unless that block explicitly returns null).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r567178516
##########
File path: spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetReadMetadataColumns.java
##########
@@ -77,6 +77,9 @@
private static final int NUM_ROWS = 1000;
private static final List<InternalRow> DATA_ROWS;
private static final List<InternalRow> EXPECTED_ROWS;
+ // We filter for $"data" notStartsWith str1, along the row group splits, to ensure the row group
Review comment:
This test suite tests the metadata columns, which is unrelated to filtering. I don't think that this should be changed.
It looks like you want a Spark test to validate the end-to-end behavior. I think another suite, or maybe a new one, would be more appropriate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-998349282
> Overall, the tests and implementation all look correct to me. I think there are a few minor things we could do, but I'm ready to commit this.
I'll backport this to Spark 3.1 and 3.0 after we've merged then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772659284
##########
File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java
##########
@@ -340,6 +340,56 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_MIGHT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int pos = Accessors.toPosition(ref.accessor());
+ PartitionFieldSummary fieldStats = stats.get(pos);
+
+ // Iceberg does not implement SQL 3-boolean logic. Therefore, for all null values, we have decided to
+ // return ROWS_MIGHT_MATCH in order to allow the query engine to further evaluate this partition, as
+ // null does not start with any non-null value.
+ if (fieldStats.containsNull() || fieldStats.lowerBound() == null) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ ByteBuffer prefixAsBytes = lit.toByteBuffer();
+
+ Comparator<ByteBuffer> comparator = Comparators.unsignedBytes();
+
+ ByteBuffer lower = fieldStats.lowerBound();
+
+ // notStartsWith will match unless all values must start with the prefix. this happens when the lower and upper
+ // bounds both start with the prefix.
+ if (lower != null) {
Review comment:
We may want to check both lower and upper here. I'd also move the prefix handling into the block, after we know that `fieldStats.lowerBound()` and `fieldStats.upperBound()` are both non-null. I don't think that `lit.toByteBuffer()` is that expensive, but it seems like a good idea to move it just in case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772723793
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -377,6 +381,31 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
return ROWS_CANNOT_MATCH;
}
+ @Override
+ public <T> Boolean notStartsWith(BoundReference<T> ref, Literal<T> lit) {
+ int id = ref.fieldId();
+
+ Boolean hasNonDictPage = isFallback.get(id);
+ if (hasNonDictPage == null || hasNonDictPage) {
+ return ROWS_MIGHT_MATCH;
+ }
+
+ Set<T> dictionary = dict(id, lit.comparator());
+
+ // Allow query engine to make its own decisions regarding SQL 3-valued boolean logic.
+ if (dictionary.contains(null)) {
+ return ROWS_MIGHT_MATCH;
+ }
Review comment:
Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#discussion_r772722530
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java
##########
@@ -368,6 +368,10 @@ public Boolean or(Boolean leftResult, Boolean rightResult) {
}
Set<T> dictionary = dict(id, lit.comparator());
+ if (dictionary.contains(null)) {
Review comment:
I'll remove it 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2062: [CORE] Add in a NOT_STARTS_WITH operator with filter pushdown
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2062:
URL: https://github.com/apache/iceberg/pull/2062#issuecomment-984024388
@cccs-eric, this is something we should complete. I didn't realize we hadn't finished it. @kbendick can you update this when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org